A World Map of LLM Answers Is an Eval, Not a Belief System
The llm-beliefs project asks models whether censorship exists in each country, then maps the answers. It is a useful reminder that simple political questions can expose model behavior, but not cleanly separate knowledge, refusal policy, prompt sensitivity, and dataset residue.
TL;DR: A yes/no map of LLM answers is useful as a behavior probe, but it should not be treated as a clean readout of what a model “believes.”
What does a map of LLM yes/no answers really measure?
The primary source here is p-e-w’s GitHub project, p-e-w/llm-beliefs, shared on r/LocalLLaMA as “Does censorship exist in [country]?, according to various LLMs.” According to p-e-w, the program asks the same yes/no question about every country to a language model, then generates an SVG world map showing the model’s responses.
That is a clever little eval.
It is also a trap if readers take the output too literally.
A question like “Does censorship exist in [country]?” sounds simple. It is not. The word “censorship” can mean state media control, platform takedowns, wartime restrictions, informal pressure, legal limits on hate speech, bans on foreign apps, or internet shutdowns. A model forced into yes/no has to compress all of that into one bit.
So the map is not measuring one thing. It is probably mixing at least four things: what the model saw during training, how the instruction tuning shaped politically sensitive answers, whether the prompt triggers safety behavior, and how willing the model is to make a categorical judgment without qualifiers.
That last part matters. Some models are trained to hedge on political claims. Others will answer bluntly. On a map, both behaviors look like geopolitical knowledge. They may actually be response style.
Why is this still useful?
Because cheap probes catch weird behavior fast.
I like this kind of project because it takes a vague concern, “models have political bias,” and turns it into something you can run, compare, and inspect. Not perfect. Not definitive. But repeatable.
The useful part is not any single country’s color. It is the pattern across models, prompts, and sampling settings. If one model says yes for nearly everyone, another refuses half the time, and another gives a suspiciously neat geopolitical split, that tells you something about product risk.
This is especially relevant for teams building search, education, research, policy, compliance, or customer support tools. Users ask politically loaded questions in plain language. They do not say, “Please distinguish between formal state censorship, private moderation, and constitutional speech limits.” They ask the messy version.
A model that turns messy questions into overconfident binaries can create a bad product moment. A model that refuses normal civic questions can also create a bad product moment. The hard part is not making the model “neutral.” The hard part is making it useful, calibrated, and honest about ambiguity.
What would make this a stronger eval?
The next version should test stability.
Ask the same question several ways: “Does censorship exist in X?”, “Are there government restrictions on media in X?”, “Is online speech restricted in X?”, “Can journalists freely criticize the government in X?” Then compare answers.
Run each prompt multiple times if sampling is enabled. Track flips. A country that changes from yes to no depending on wording is more interesting than a country marked one color on one pass.
Add a third answer: uncertain. Better yet, ask for a short rationale and citeable basis, then score the rationale separately from the label. The binary map is good for noticing patterns. The explanation is where you find whether the model is reasoning, parroting, refusing, or laundering a stereotype.
I would also separate model families and deployment contexts. A local open-weight model, a hosted chat product, and an API with safety layers may behave differently for reasons that have nothing to do with the base model’s stored knowledge. If you are comparing them, say exactly what you ran.
For builders, I would steal the shape of this eval, not the exact topic. Pick 20 to 100 sensitive questions your product will actually face. Turn them into a small regression suite. Run them across candidate models and prompt versions. Look for refusals, false certainty, unstable answers, and policy surprises. The catch most readers miss: the map is not the verdict. The diff between runs is the product signal.