Grounded Entity Biographies: A Different Fix for Long-Video Memory

TL;DR: A paper called “Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies” argues that retrieving the right moments from a long video is not enough, and that tracking the identity of specific physical objects across those moments is what actually answers hard questions, hitting 72.0% on EgoLifeQA, 4.4 points above the prior best.

Most long-context and memory work treats the timeline as the unit. Chunk the video, describe each chunk, retrieve the relevant chunks, feed them to the model. The GEB paper (arXiv cs.AI / cs.CL / cs.LG, same work cross-listed) makes a quieter but sharper claim: the timeline is the wrong index. What you often need is the object, followed across the whole timeline.

What problem is GEB actually solving?

Picture a week of egocentric footage. Someone asks where you left a specific mug, or which of two similar bags had your keys. A standard retrieval memory pulls back events that mention “mug” or “bag.” That is the trap the authors point at directly: different objects can share the same description, and separate observations of the same object stay disconnected across events. So you retrieve text that matches the words in the question, and you still cannot resolve which physical thing the question is about.

The authors call this the gap between events and identity. Retrieving relevant events “does not necessarily recover the biography of the particular entity a question concerns.” That framing is the whole insight. A chronological description tells you what happened. It does not tell you that the object in clip 3 and the object in clip 190 are the same object.

Grounded Entity Biographies fix this by grouping visually grounded observations of the same physical instance across clips into one retrievable record, a “biography” for that object, while keeping the context of each moment attached. At question time the model gets two things: the biography of the entity in question, plus the episodic evidence around it. The identity links were established when the memory was built, not guessed at query time.

That build-time versus query-time distinction is the part builders should sit with. Most RAG systems do all their reasoning at retrieval. GEB pushes the expensive association work, deciding that these pixels and those pixels are the same object, into memory construction. Retrieval then just follows links that already exist.

Why does identity-first memory beat better descriptions?

The obvious objection: why not write richer captions? If descriptions are ambiguous, describe harder. The authors ran the ablation and answered it. Additional descriptions alone do not fully recover the gains. Both grounded identity association and biography reading contribute independently.

That result is worth more than the headline number. It says the win is not “more text about each moment.” It is a structural change in how memory is organized. You can pile description on top of a timeline forever and still fail to link two sightings of the same physical object, because language about appearance is lossy and objects look alike. Visual grounding, tying an entity to actual pixels across clips, carries information that captions drop.

This is the same lesson that keeps showing up in agent memory generally. Flat retrieval over a growing log degrades as the log grows, because relevance by text similarity is not the same as relevance by reference. GEB is a video-specific instance of a broader pattern: memory works better when it is organized around the entities a system reasons about, not just the timestamps at which things were logged.

How strong is the evidence?

Four benchmarks, including day-long and week-long recordings, with improvements in both multiple-choice and open-ended question answering. The named result is 72.0% on EgoLifeQA, 4.4 percentage points above the best published result. The ablations back the mechanism rather than just the score.

I want to be honest about what the sources here do and do not give. All three entries are the same paper cross-listed across arXiv categories, so this is one result, not three independent confirmations. The abstract reports the numbers; I have not seen an external replication, and 4.4 points on a single benchmark is a real gain but not a landslide. The claim I would treat as solid is the qualitative one: identity association adds something descriptions do not. The claim I would hold more loosely is exactly how much it generalizes past egocentric video benchmarks to, say, security footage, sports, or multi-camera setups where the same object appears from wildly different angles.

The hard part GEB depends on is also its weakest seam: getting the identity links right at build time. Re-identifying the same physical object across hours of footage, lighting changes, occlusion, and near-duplicates is exactly the computer vision problem that has resisted clean solutions for years. The paper reports it works well enough to lift accuracy. It does not, in this abstract, tell us how the links degrade when re-identification is wrong, or how much compute the construction phase costs. Those are the questions I would want answered before betting a product on it.

What can a builder take from this now?

Even without the video use case, the design pattern transfers. If you are building any long-horizon agent memory, RAG over months of chat, customer histories, document sets that reference the same entities repeatedly, the GEB move is to stop indexing purely by chunk and start maintaining entity records that accumulate observations over time. When a query is about a specific customer, contract, or product, retrieve that entity’s accumulated record alongside the relevant episodes, not just the episodes that lexically match.

The concrete experiment: take your existing retrieval memory and add an entity layer. Resolve mentions to canonical entities at ingest time, attach each observation to the resolved entity, and at query time pull the entity’s full record plus the top matching chunks. Measure whether questions that require tracking one thing across many events improve. Those are the queries flat retrieval fails silently on, because it returns plausible-looking chunks and no one notices the answer is about the wrong instance.

The catch most readers will miss: the entire value depends on entity resolution being correct at build time, and resolution errors are worse than retrieval misses because they are invisible. A bad chunk retrieval gives you an obviously irrelevant passage. A bad identity link gives you a confident answer about the wrong object. GEB moves the hard problem upstream, which is the right move, but it also concentrates the risk there. Build the entity layer, and spend your evaluation budget on how often the links are wrong, not just on how often the final answer is right.