Compliance LLMs need different workflows for passports and DPIAs

TL;DR: LLMs look useful for compliance paperwork, but the workflow has to fit the artifact: strict schemas need validation, loose assessments need context and expert review.

What did the paper actually test?

The primary source here is “From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry”, published on arXiv cs.AI. It looks at a very practical problem: companies are being asked to produce regulatory documents that depend on messy internal data, supplier records, legal interpretation, and domain expertise.

Two examples matter.

The EU’s Ecodesign for Sustainable Products Regulation introduces Digital Product Passports for life cycle transparency. In the battery context, the paper discusses Digital Battery Passports, which have stricter formatting expectations. GDPR, by contrast, requires Data Protection Impact Assessments, but DPIAs do not follow one standard format and need privacy, legal, technical, and organizational judgment.

That distinction is the useful part. This is not another generic “LLMs can automate compliance” claim. The researchers tested how data extraction instructions and regulatory vagueness affect LLM-produced compliance artifacts, then benchmarked model outputs against manually created ground-truth schemas.

Their finding is intuitive, but important: stricter artifact formats produced more consistent results across prompt contexts, while looser DPIA-style artifacts needed more context in the prompt to stay complete and consistent. The catch: strict formatting did not eliminate risk. The paper reports that stricter Digital Battery Passport-style guidelines can still lead to more hallucinations in the output.

That should sound familiar to anyone who has built with LLMs. Structure improves shape. It does not guarantee truth.

Why do strict schemas still hallucinate?

Because a schema can tell a model where an answer belongs, not whether the answer exists.

For a Digital Battery Passport, a system might know that a field is required, that the answer should be formatted a certain way, and that missing values look bad. If the upstream data is scattered across suppliers, PDFs, ERP exports, spreadsheets, and product systems, the model may still be forced into a bad choice: leave a hole, infer too much, or invent a plausible value.

That is the compliance danger zone. A clean artifact can create false confidence. The better the formatting, the easier it is for a reviewer to miss unsupported content.

So the builder lesson is not “use bigger prompts.” It is more specific: separate extraction, transformation, evidence, and generation. For structured passports, the LLM should not be the system of record. It should map source data into fields, cite where each value came from, flag missing data, and pass the result through deterministic validators. If a required field has no source, the right output is not a polished guess. It is an exception.

The paper’s result on stricter guidelines is a useful warning. Consistency is not the same as accuracy. In compliance, consistency without provenance can be worse than messy output, because it scales bad confidence.

Why are DPIAs a different problem?

DPIAs are less like form filling and more like structured reasoning under ambiguity.

GDPR says organizations need to assess privacy risks in certain contexts, but DPIAs do not have one universal template. A good DPIA depends on system design, data flows, user groups, risk scenarios, mitigations, and institutional policy. The arXiv paper reports that less strict DPIA formatting required higher context prompts to maintain consistency and completeness.

That tracks. If the task is ambiguous, the prompt has to carry more of the organization’s reality. What data is collected? Who processes it? Where does it move? What safeguards exist? Which risks are already known? Which teams own mitigations?

The failure mode here is not only hallucination. It is omission. A model can write a reasonable-looking DPIA that misses the hard part, like downstream data sharing, retention, vulnerable users, or a mitigation that exists in policy but not in production.

I would not treat an LLM-generated DPIA as a compliance artifact. I would treat it as a drafting interface for experts. Useful for first passes, gap spotting, red-team questions, and consistency checks across projects. Not a substitute for privacy review.

For builders, the practical move is to split the product in two. For passports, build around schemas, source links, missing-field queues, and validators. For DPIAs, build around context gathering, interview-style intake, risk libraries, internal policy retrieval, and reviewer sign-off. The catch most readers miss: the LLM is not the compliance layer. The compliance layer is the evidence trail, the validation logic, and the accountable human workflow around it.