Dutch government LLMs need trade-off tables, not model rankings

TL;DR: For government LLM use, the useful question is not “which model is best,” it is “which trade-offs are acceptable for this public service, in this language, under these constraints.”

What does a government LLM benchmark need to measure?

The primary source here is the arXiv paper “From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch.” It is aimed at a gap that keeps showing up in real deployments: public agencies are trying to use LLMs, but most evaluation work still looks too generic, too English-first, and too detached from public administration.

The paper presents the “Grip on LLMs” framework, built with domain experts from a major Dutch municipal organisation. That matters. A civil-servant chatbot is not the same problem as a coding assistant or a consumer writing tool. If a resident asks about a permit, benefit, tax issue, or local rule, the model has to be right, has to admit uncertainty, has to avoid discriminatory behavior, and has to fit public budget and sustainability goals.

The framework turns that into six evaluation dimensions: factuality, honesty, social bias, energy consumption, cost, and training data transparency. That list is more practical than the usual leaderboard bundle. It includes things engineers can test, things procurement teams care about, and things policymakers will be asked to defend later.

The paper reports results across more than 30 multilingual and Dutch-specific models. The clean finding: no single model wins across all dimensions. Higher quality consistently comes with greater environmental impact and financial cost. Bias, meanwhile, remains largely independent of both. Paying more or using a bigger model does not automatically buy your way out of bias.

Why separate factuality from honesty?

This is the most useful distinction in the paper.

Factuality means the model answers correctly. Honesty means the model acknowledges what it does not know. The Grip on LLMs results find these are governed by distinct properties. High factuality does not imply high honesty.

That is a big deal for public-sector workflows. Many model evals reward answer accuracy and stop there. But a government assistant that is usually right and confidently wrong in edge cases can be worse than a slightly less capable system that knows when to escalate.

Think about a municipal chatbot answering a question about eligibility. If the answer is wrong, that is bad. If the answer is wrong and delivered with authority, that is worse. It can change what a resident does next. It can also create an audit problem for the agency.

This is where “AI safety” gets very concrete. Not abstract alignment. Not sci-fi. A model should say “I don’t know,” “I need more information,” or “contact the municipality” when the policy boundary is unclear. That behavior needs to be tested separately from raw correctness.

For builders, this suggests a different acceptance test. Do not only run a golden set of answerable questions. Include unanswerable questions, ambiguous questions, out-of-scope questions, and questions where the right response is refusal or escalation. Then score those independently.

Why does Dutch matter here?

Non-English evaluation is not a side quest. It is the product.

Government services operate in the language residents use. Dutch-specific performance matters for tone, policy terms, legal phrasing, forms, and local administrative concepts. A model that performs well on English benchmarks may still be brittle in a Dutch municipal context.

The Grip on LLMs framework is interesting because it links language, values, and deployment constraints in one evaluation suite. It does not treat Dutch as a translation layer pasted on top of an English benchmark. It starts from the actual environment: civil servants, municipal needs, stakeholder priorities, and chatbot users.

The public-facing model overview is also important. Model selection in government is not only an engineering decision. Procurement, legal, sustainability, policy, service design, and leadership all get a vote. If the evaluation artifact is only readable by ML engineers, the real decision will happen somewhere else, probably with less evidence.

This is the part I would copy. Not the exact Dutch benchmark, unless you work in that context. The operating model. Start with stakeholder values. Turn them into measurable dimensions. Test multiple models. Publish the trade-offs in a format non-technical decision-makers can actually use.

If you are building for a public agency, or any regulated service, do not ask vendors for one benchmark score. Ask for a trade-off sheet. Test factuality and honesty separately. Add cost and energy before the pilot, not after procurement. And localize the eval to the language and policies your users actually face. The catch most teams miss: the “best” model is often only best after you quietly drop the constraints that made the deployment hard in the first place.