Muon’s agent RL win is real, narrow, and stack-dependent
A small ALFWorld study suggests Muon can materially improve sparse-reward agent training, but only when the optimizer, advantage estimator, and learning rate line up. The useful lesson is less “switch to Muon” than “tune the RL stack as a coupled system.”
Muon keeps showing up as one of those optimizer ideas that builders want to reduce to a slogan. “Better than AdamW” is the tempting version. The new arXiv study, When Does Muon Help Agentic Reinforcement Learning?, makes that slogan harder to defend, in a useful way.
The researchers tested vanilla Muon against AdamW in sparse-reward agentic RL on ALFWorld, using Qwen2.5-0.5B-Instruct. This is not a giant benchmark sweep. It is matched single-seed work, which means the right posture is interest, not victory lap. Still, the numbers are big enough to pay attention to.
Under Group-in-Group Policy Optimization, applying Muon only to hidden weight matrices raised final-window validation success from 0.290 to 0.546. That is an 88% relative jump. High-rate AdamW controls kept no post-update success. In another setup, at a 3e-5 learning rate, Muon improved GRPO from 0.161 to 0.268. At 1e-5 with GraphGPO, Muon reached 0.901, improved normalized validation AUC from 0.399 to 0.556, and hit 0.5 and 0.75 success 30 and 60 updates earlier.
Those are not tiny deltas. They are also not a general law.
The optimizer is not the whole treatment
The interesting part is not just that Muon won. It is where it won.
The study’s strongest result comes from a very specific recipe: sparse-reward agent tasks, Qwen2.5-0.5B-Instruct, ALFWorld, certain policy optimization methods, certain learning rates, and Muon applied only to hidden weight matrices. Change the advantage estimator or learning rate, and the effect changes.
That matters because RL post-training is already a pile of interacting choices. The optimizer changes the shape of the update. The advantage estimator decides what signal gets amplified. The learning rate decides how violently the model moves. In sparse-reward settings, where most attempts produce little useful feedback, those pieces can either cooperate or cancel each other out.
The GraphGPO result shows this clearly. At 3e-5, the late-window gap between Muon and AdamW narrows near saturation. At 1e-5, Muon looks much more useful, reaching higher success and getting there earlier. That is not “Muon is better.” That is “Muon changes the viable tuning region.”
For builders, that distinction is the product.
Sparse-reward agents may expose optimizer differences faster
Pre-training often hides optimizer differences under massive data, long runs, and averaging effects. Agentic RL does not always give you that cushion. ALFWorld-style tasks are brittle. The agent has to interact, make multi-step choices, and eventually get a sparse success signal. Bad updates can erase useful behavior quickly. Good updates can compound.
That makes this kind of experiment valuable, even if it is early. It suggests optimizer choice may matter more in post-training agents than many teams assume, especially when the environment gives weak feedback and the policy has to preserve language competence while learning action behavior.
But the study also says the quiet part clearly: multi-seed and cross-task validation remain open. Single-seed comparisons can mislead. ALFWorld is one environment family. Qwen2.5-0.5B-Instruct is one base model size and type. We should not turn this into a universal training recipe yet.
The practical read is narrower and better: if you are doing agent RL and AdamW feels unstable or slow under sparse rewards, Muon deserves a controlled trial. Not a blind swap. A controlled trial.
Start with the same model, same environment, same reward, and same policy optimizer. Run AdamW and Muon side by side. Apply Muon only where the study found signal, hidden weight matrices, before expanding scope. Sweep learning rate around your current setting rather than assuming the AdamW rate transfers. Track not just final success, but AUC and time-to-threshold, because earlier useful behavior can matter more than a final leaderboard number. The catch most readers miss: Muon is not a magic agent trainer. It is another knob in a coupled RL system, and its value shows up only when the rest of the stack lets it.