When the Teacher Knows to Quit: RetireOPD and Distillation for Agents

TL;DR: RetireOPD trains a small agent with reinforcement learning plus a self-teacher, then has the student drop that teacher automatically once the extra supervision stops helping, and that “know when to quit” trick beats plain RL by 11 to 19 points on standard agent benchmarks.

The paper is “RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning,” posted to arXiv (cs.AI and cs.CL). It targets a very specific and very real problem: training multi-turn agents with RL is starved for signal, and the usual fixes quietly break in agentic settings. What makes this one worth reading is not a new architecture. It is a scheduling insight. The student decides when the teacher has nothing left to teach.

Why is reinforcement learning so hard for multi-turn agents?

Here is the core pain. When you train an agent with RL across a long task, the model takes dozens of actions, and at the end it gets one number. Success or failure. Maybe a partial score. That single scalar reward has to explain everything that happened across the whole trajectory. Which token in step three was smart? Which decision in step nine sank the run? The reward does not say. This is the credit assignment problem, and it gets worse the longer the task runs.

The standard workaround is on-policy distillation, or OPD. You build a teacher that has some advantage the student lacks, in this case “privileged task skills,” and you let the teacher supply dense, token-by-token supervision. Instead of one number at the end, the student gets guidance at every step. That is the appeal. Turn a sparse signal into a rich one.

RetireOPD’s authors flag two things that go wrong with this recipe in agentic tasks. First, privileged information does not automatically make a teacher reliable. Knowing the answer key does not mean you teach well. Second, and this is the sharper point, the value of teacher supervision is stage-dependent. Early on, the teacher helps a lot. Later, the student has internalized the useful behavior, and the teacher becomes noise, or worse, a ceiling.

How does the “self-retiring” part actually work?

Most distillation setups use a fixed schedule. Distill for N steps, then stop, or blend teacher and RL signal on some hand-tuned curve. That schedule is a guess, and a bad guess costs you either undercooked skills or a student stuck imitating a teacher it has already outgrown.

RetireOPD replaces the guess with a rule the student applies to itself. The setup has two phases. First, they optimize a decoupled, skill-conditioned teacher using environment rewards. The teacher is trained separately, conditioned on the skills, rather than being the same frozen checkpoint doing double duty. Then they train a skill-free student jointly with both RL and OPD, so the student learns from environment reward and from the teacher at the same time.

The retirement is the interesting bit. They call it Adaptive Retirement. The student drops the teacher on its own once two conditions hold: the discrepancy between student and teacher stops shrinking, and the student reaches a target fraction of the teacher’s success rate. Once both are true, training continues with RL alone. No predefined distillation schedule. The student essentially says “I have learned what you had to give, and I am no longer closing the gap, so we are done here,” and then keeps improving on its own reward signal.

That framing matters because it addresses the second failure mode directly. If teacher supervision is stage-dependent, then the right move is not a better fixed schedule. It is a signal that detects the stage you are in. Discrepancy plateau plus a success-rate threshold is a cheap, sensible proxy for “the teacher has stopped adding value.”

What do the numbers actually show?

The evaluation runs on Qwen2.5 models from 1.5B to 7B, tested on ALFWorld and WebShop, two staple agent benchmarks. ALFWorld is embodied household task-solving in text. WebShop is a simulated shopping environment where the agent has to find and buy the right product from a description.

Against an RL baseline, RetireOPD reports a 14.1% to 18.8% improvement on ALFWorld success rate and an 11.8% to 19.0% improvement on WebShop accuracy. The range spans the model sizes. The single most telling result is buried in the last line of the abstract: the student surpasses its own skill-conditioned teacher in every setting.

Read that again. The thing it was learning from ends up worse than the thing it produced. That is exactly what you would predict if the “retire the teacher” thesis is correct. A student that kept imitating its teacher forever could not beat it. A student that uses the teacher as a bootstrap and then walks away on RL can. The teacher is scaffolding, not a ceiling, and the retirement mechanism is what turns one into the other.

A few honest caveats. These are small models, 1.5B to 7B, and two benchmarks. ALFWorld and WebShop are well-worn, which is good for comparability and bad for surprise, since methods can quietly overfit to familiar environments. The gains are real and consistent across sizes, but I would want to see this on longer-horizon, messier tasks before calling it a general recipe. The abstract also does not detail compute cost of training the decoupled teacher, and a two-phase setup is more moving parts than plain RL. Worth knowing before you budget a run.

Who should care about this, and why now?

The people who should care are anyone training their own agent on a small open model instead of calling a frontier API. That population is growing, because a 7B agent that actually completes tasks is cheaper to run, easier to host locally, and yours to control. The bottleneck for those teams has never been the base model. It is getting a useful learning signal out of long, sparse tasks without burning a fortune on RL rollouts.

RetireOPD is attractive there precisely because it does not require a bigger model or a better environment. It requires a smarter schedule, and the schedule tunes itself. The teacher is built from your own system, so there is no dependency on a proprietary larger model to distill from. That keeps the whole loop inside your control, which matters if you are working with open weights on purpose.

Practitioner’s take: if you already run agentic RL on small models, the cheapest experiment is to steal the Adaptive Retirement idea alone, even before adopting the full two-phase teacher. Instrument your existing distillation with two metrics, student-teacher discrepancy and student success as a fraction of the teacher’s, and cut the teacher off when the discrepancy plateaus and you hit your threshold, then let RL run out the rest. That is a small code change that tests the paper’s central claim on your own tasks. The catch most people will miss: the win depends on your teacher being a genuine floor, not a ceiling, so if your “teacher” is a much stronger frontier model, retiring it early will hurt, not help. This method assumes a self-teacher the student can plausibly surpass. Match the technique to that shape or it will not reproduce.