LLM personas fail when opinions have to change
The arXiv paper Before You Poll with LLMs tests whether synthetic survey respondents update beliefs like humans after seeing balanced information. The answer is no, and the failure mode matters for anyone using LLMs as stand-ins for customers, voters, or employees.
TL;DR: LLM personas can mimic static opinions, but the arXiv paper “Before You Poll with LLMs: A Deliberative Diagnostic Framework” shows they break when asked to update those opinions after new information.
What does this test that normal synthetic polling misses?
Synthetic polling has a simple appeal. Give a model a persona. Ask it survey questions. Scale that to hundreds or thousands of simulated people. Cheap, fast, directional.
The problem is that most checks ask whether the model starts in the right place. Does a suburban Republican dad sound like a suburban Republican dad? Does a young progressive renter answer like a young progressive renter? That can be useful, but it is a static snapshot.
“Before You Poll with LLMs: A Deliberative Diagnostic Framework” argues the harder test is dynamic fidelity: whether simulated people change their beliefs like real people after seeing the same new information. That matters because a lot of real opinion research is not just about what people think before deliberation. It is about what moves them.
The paper grounds its test in deliberative polling and uses data from America in One Room: 526 personas, 72 questions, and identical informational interventions given to humans and LLM personas. The diagnostic compares belief shifts, not just initial answers.
That distinction is the whole story. A model can produce a plausible partisan response and still misrepresent how that same person would react after reading balanced arguments.
What failed across the models?
According to “Before You Poll with LLMs,” every tested frontier model failed, but not in the same way.
GPT-5.1 showed what the paper calls reversal. Its personas became more hostile toward the opposing party after balanced information, while humans became less hostile. The paper reports this reversal was selective: 80% on outgroup questions versus 26% on policy questions, and symmetric across partisan identities.
Gemini 2.0 Flash, Claude Sonnet 4.5, and Llama 3.3 70B moved in the right direction, but too far. The paper calls this overshoot, with shifts at 5 to 7 times the human magnitude.
DeepSeek V3 barely moved. The paper calls that rigidity.
That spread is useful. This is not one generic “LLMs are biased” result. It is a map of different failure shapes. One model amplifies identity conflict. Others overreact to persuasion. Another acts like no argument matters. If you are using synthetic respondents to test messaging, policy language, employee sentiment, or product positioning, those are not small errors. They point in different operational directions.
The paper’s phrase for the broader pattern is self-sycophancy: the model conforms to its internal stereotype of the persona instead of reasoning from the information provided. I like that term because it names something builders see all the time. The model does not just flatter the user. It flatters the character sheet.
So are LLM personas useless for research?
No. But they need to be treated as instruments with calibration limits, not as cheap humans.
For static exploration, LLM personas can still help brainstorm objections, surface likely language, and stress-test survey design. I would not throw that away. The mistake is using that output as if it were measured public opinion, especially when the question depends on persuasion, deliberation, or belief change.
The diagnostic in “Before You Poll with LLMs” is a better bar: expose humans and model personas to the same information, then compare the direction and size of the shifts. If the model overshoots by 7x, freezes, or reverses on identity-loaded questions, you have learned something before the bad deck goes to leadership.
There is also a broader eval lesson here. Many AI benchmarks still reward the first answer. Real workflows often depend on the second state: after correction, after evidence, after disagreement, after context changes. Opinion simulation is just the cleanest place to see the bug.
Practitioners should try LLM personas as a draft layer, not a measurement layer: use them to generate hypotheses, then run a small real-human check on anything that changes budget, product direction, policy, or messaging. The catch most teams miss is that “sounds like the target audience” is not the same as “moves like the target audience.”