Declarative attention lets models skip the context they do not need

TL;DR: Declarative Attention is a simple inference trick: ask the model where it needs to look, then skip most of the KV cache when it does not need the whole context.

What is Declarative Attention actually doing?

The primary source here is the arXiv paper “Language Models Can Control Their Own Attention,” listed under cs.AI and cs.CL. The core claim is clean: language models often need only a small slice of a long context, but standard global attention still reads across the whole KV cache during decoding.

That matters because long-context use is no longer theoretical. People are stuffing models with giant chats, codebases, policy docs, support histories, legal records, and meeting archives. The model may only need one old paragraph, but the inference engine still pays to scan the whole cached context token after token.

Declarative Attention, or DA, changes the interface between the model and the inference engine. Instead of an outside scoring system deciding what might be relevant, DA asks the model to declare its own attention mode during generation.

The paper describes three modes: full context, a specific region, and recent output only. The inference engine parses these declarations like tool calls. If the model says it only needs a region, the engine avoids reading most of the KV cache. If it says recent output only, it can skip even more.

This is interesting because it treats attention control as a protocol problem, not only a model architecture problem. The model already has some sense of what it is trying to retrieve. DA turns that hidden preference into an executable instruction.

Does this make million-token context cheap?

Not by itself.

The paper reports zero-shot tests across 15 long-context tasks on off-the-shelf models, Gemma-4-31B and Qwen-3.6-27B. DA reduced total attended tokens during decoding by 52.0% for Gemma-4-31B and 31.1% for Qwen-3.6-27B. The reported accuracy drops were 1.27 percentage points and 2.75 percentage points, and the paper says those drops shrink with model scale.

Those are useful numbers. They are not a blank check.

First, DA attacks decoding attention cost. It does not erase the cost of ingesting the original long prompt. If your workflow is dominated by prefill, document parsing, embedding, retrieval, or tool calls, this will not magically halve the whole bill.

Second, the method depends on the model making good attention declarations. If the model declares the wrong region, it may miss the evidence. That failure mode is familiar from retrieval-augmented generation, except here the retrieval decision is coming from inside the model’s generation protocol.

Third, this is zero-shot evaluation on two named model families. That is promising, but still early. The paper itself points to training-based methods as future work, which is probably where this gets more serious. A model trained to emit reliable attention declarations could behave differently from a prompted off-the-shelf model.

Why should builders care?

Because most long-context products are wasteful in a very boring way.

A customer support agent does not need every token of a three-year account history to answer whether a refund was approved last week. A code assistant does not need the whole repo for every line of a patch. A research assistant does not need to scan the full document bundle while drafting boilerplate transition sentences.

Today, builders usually handle that with retrieval, chunking, summarization, or context pruning. DA suggests another layer: once the context is already loaded, let the model tell the runtime which parts it wants during each phase of the answer.

That could pair well with agentic workflows. An agent might use full context when planning, a specific region when quoting evidence, and recent output only when formatting the final response. Same loaded context. Different attention budget at different moments.

The catch is product reliability. Users do not care that you saved attention reads if the model skipped the paragraph that mattered. So I would not start with high-stakes legal, medical, or compliance workflows. I would test this first where the answer can be checked, where citations are required, and where latency or serving cost is painful enough to justify a new inference path.

Practitioner’s take: if you run long-context workloads, log which parts of the context your model actually uses, then compare that against DA-style declared regions. You do not need to rewrite your stack tomorrow. Build an eval set with long prompts, required evidence spans, answer quality, and latency. The missed detail rate is the number to watch. Savings are only real if the model still looks in the right place.