Language critiques are a better training signal than a score, if you can afford them
A new imitation learning paper argues that language critiques can carry richer supervision than scalar scores when demos are imperfect, with promising results across control tasks and some open questions about label creation, evaluation detail, and whether builders can make the feedback cheap enough.
Suboptimal demos are everywhere.
A robot arm almost grasps the object, then bumps it. A navigation agent takes a path that works, but wastes time. A gameplay policy wins one encounter because the opponent makes a mistake, not because the policy learned the right tactic. Most imitation learning pipelines have to turn that mess into a training signal.
The usual move is compression: confidence estimates, discriminator scores, importance weights. Basically, “this part was good” or “this part was bad,” squeezed into a number.
The Language-Critique Imitation Learning paper argues that this throws away the useful part. A scalar cannot say, “you’re close to the target, but you turned too early,” or “the grasp is aligned, but the approach angle is wrong.” That kind of feedback is structured. It has progress, diagnosis, and correction in one package.
The interesting shift is not language as interface, it is language as loss
Two arXiv listings point to the same work, one under cs.AI and one under cs.LG. The claim is narrow and worth paying attention to: use natural language critiques as supervision for policies trained from imperfect demonstrations.
The method first creates language labels from demonstrations. Those labels describe current task progress, identify suboptimal behavior, and give corrective guidance. Then the authors train policies with a language-critique loss, rather than reducing the critique back into a scalar score.
That last bit matters. If the language is just a prettier wrapper around a reward number, this is mostly product design. The paper’s bet is that the policy can learn from the internal structure of the critique itself.
They instantiate the idea in two familiar settings: behavior cloning and diffusion policies, producing LC-BC and LC-DP. The reported evaluations cover continuous control tasks across navigation, manipulation, and gameplay, and the paper says the language-critique versions consistently beat strong imitation learning and offline reinforcement learning baselines.
The result fits a bigger pattern in applied AI
We keep seeing the same lesson across agents, coding tools, tutoring systems, and now control policies: rich feedback beats thin feedback when the model can use it.
A pass/fail label can train behavior. A critique can train judgment.
This is why code models get better from review comments, not just green or red tests. It is why agent traces with explanations can be more useful than outcome-only logs. And it is why imitation learning from imperfect demos has always felt slightly under-instrumented. The demonstrator often contains signal even when the action is wrong. The mistake has shape.
The paper also includes a theoretical result: its objective upper-bounds the expert performance gap under standard assumptions. Useful, but I would not overread it without the full setup. Theory in these papers often clarifies what has to be true for the method to work. It does not prove the method will be easy to run in your warehouse, game engine, browser agent, or kitchen robot.
The practical questions are still open from the abstract alone. How are the critiques generated? How much human labor is needed? Are the labels produced by another model? If so, how brittle are they when the critic is wrong? And when the tasks get long-horizon, does language still give fine enough supervision, or does it become vague narration?
That is the catch. Language is higher bandwidth than a score, but it is also messier. You can train on “move left sooner” if that phrase maps cleanly to state and action. You can get garbage if the critique is generic, late, contradictory, or detached from the control problem.
For builders, I’d treat this as a design pattern before treating it as a plug-in method. If you have logs of failed or mediocre actions, don’t only label them good or bad. Add compact critiques: what was achieved, what went wrong, what should change next. Start with one narrow workflow where failure modes repeat. Then compare scalar-only training against critique-augmented training. The missed catch: the critic becomes part of the system. If critique quality is cheap and consistent, this could be a real advantage. If critique quality is expensive or noisy, the score may still win.