ViSkill and the case for keeping agent memory visual

TL;DR: When your agent works in a visual or spatial world, storing its learned skills as text throws away the geometry that made them work, and a new paper shows you get better results keeping those skills as images the model can read directly.

Most agent memory systems do the same thing. The agent does something that works, you summarize that success in words, you stash the words, and later you retrieve them to guide the next attempt. It is clean. It is also lossy, especially when the task is about where things are in space rather than what they are called.

That is the bet behind ViSkill, a paper out of the ZJU-REAL group titled “ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills” (arXiv cs.CL, code at github.com/ZJU-REAL/ViSkill). The authors argue that text-centric skill memory “linearizes spatial layouts and action-state correspondences into language that loses critical geometric structure.” Put plainly: when you describe a grid puzzle in a sentence, you are describing a picture of a grid in words, and the model has to reconstruct the picture to use the advice. Why not just keep the picture.

What is ViSkill actually doing differently?

The core move is a “composite visual skill card.” Instead of distilling a successful trajectory into a text note, ViSkill encodes it as a visual artifact the vision-language model can look at directly. Those cards go into a skill library. When the agent faces a new situation, it retrieves relevant cards, and those cards shape two things at once: inference (what the agent decides to do) and reward (how the training signal gets shaped during optimization).

The second move matters more than it sounds. Most skill-augmented agent work treats skill-building and policy training as two separate jobs. You collect skills over here, you train the policy over there, and the two never really talk. ViSkill wires them into one loop. Successful trajectories get distilled back into the library, the library feeds the next round of training, and the paper describes this as a “closed feedback loop in which skill accumulation and policy improvement reinforce each other.” Better skills make a better policy, a better policy produces better trajectories, better trajectories become better skills.

There is also an optional cold-start mechanism to get the early stages moving, since an empty skill library gives you nothing to retrieve. More on why that is a real problem in a second.

Do the numbers hold up?

The paper reports an overall success rate of 0.89 across three environments, Sokoban, FrozenLake, and PrimitiveSkill, rising to 0.91 with cold-start initialization. The authors say this beats every proprietary and open-source baseline they tested and converges faster than standard PPO.

A few honest caveats before anyone gets excited. These are controlled, grid-and-gym style environments. Sokoban and FrozenLake are classic spatial reasoning testbeds, which is exactly the setting where “keep the geometry” should help most. That is both the point and the limitation. The result is a strong signal that visual-native memory beats text-native memory when the task is inherently spatial. It is not evidence that you should convert your text-based customer support agent to visual skill cards. The sources here do not test that, so I am not going to claim it.

“Converges faster than standard PPO” is the line I would poke at hardest. PPO is a reasonable baseline but a vanilla one, and faster convergence against it is a lower bar than beating a well-tuned modern RL setup. The paper does say it outperforms the other evaluated baselines on success rate too, which is the number that matters more. Still, I would want to see the full baseline list and the per-environment breakdown before treating 0.89 as a settled headline. As of the abstract, those details sit in the paper rather than in what I can confirm here.

Why does visual-native memory matter beyond puzzles?

Here is the broader idea worth taking away even if you never touch a Sokoban board.

Agent memory has a representation problem that nobody likes to talk about. We default to text because text is easy to store, easy to embed, easy to retrieve with the RAG stack everyone already has. But the format you store a skill in decides what survives. Text survives concepts and loses coordinates. If your agent operates a UI, navigates a map, manipulates a layout, inspects a chart, or reasons about physical arrangement, you are almost certainly throwing away structure every time you write a memory as a sentence.

ViSkill’s answer is to stop translating. Keep the skill in the modality the task lives in. For spatial work that is images. The deeper principle generalizes: match your memory representation to the shape of the problem, not to the convenience of your retrieval infrastructure. That is the line I would underline.

The closed loop is the other transferable idea. Separating skill collection from policy training is the default because it is operationally simpler, but it means your two learning processes never compound. Letting each feed the other is harder to engineer and, per this paper, worth it.

What is the catch for builders?

The cold-start problem is the one most people will underestimate. A skill library that reinforces itself is powerful once it has skills. On day one it has none, so retrieval returns nothing, so the agent behaves like it has no memory at all, so it generates mediocre trajectories, so your library fills with mediocre skills. Feedback loops run both directions. The ViSkill authors ship a cold-start mechanism precisely because this is not a footnote, it is the difference between the loop igniting and sputtering. If you build your own version without seeding it, expect a slow and possibly poisoned start.

The second catch is cost. Visual skill cards are not free to store or retrieve compared to text embeddings, and feeding images into a VLM at both inference and reward-shaping time means more tokens and more compute per step than a text note would. The paper’s efficiency win is in sample efficiency and convergence speed, not necessarily in per-step cost. Those are different ledgers. Decide which one you are optimizing before you copy the architecture.

Practitioner’s take: if you are building an agent for anything spatial, UI automation, robotics, map or layout tasks, game-like environments, stop reflexively summarizing successful runs into text. Try keeping a small gallery of annotated screenshots or state snapshots as retrievable memory and feed the relevant ones back in at decision time. Start with the cold-start problem solved first: seed the library with a handful of hand-made good examples before you let the agent self-improve, or the loop fills with garbage. Measure two things separately, sample efficiency (how many episodes to competence) and per-step cost (tokens and latency), because visual memory tends to win the first and lose the second. The catch most readers will miss is that this is a claim about spatial tasks specifically. For text-native work like support or research, your text memory is probably the right representation already, and switching to images would cost you more and buy you nothing.