When the Chain of Thought Doesn't Actually Drive the Answer
A new reward method called CRoS trains reasoning models by scrambling the order of their reasoning steps instead of deleting them, aiming to reward chains of thought that genuinely cause the final answer rather than decorate it.
TL;DR: A reasoning model can write a convincing chain of thought and still land on its answer for reasons that have nothing to do with that chain, and a paper called CRoS proposes a cleaner way to reward the model only when the reasoning actually drives the result.
Most of the reasoning model boom runs on one assumption: if the model shows its work and gets the right answer, the work must have helped. That assumption is shakier than it sounds. The chain of thought (CoT) you see can be partly or wholly decorative, a plausible story written alongside an answer the model arrived at some other way. When you train with rewards that only check the final answer, you have no guarantee the reasoning is load-bearing.
That is the gap “CRoS: Causality Reward on Scrambled CoTs Improves Accuracy and Reasoning Faithfulness” goes after. The paper (posted to arXiv under both cs.AI and cs.CL) proposes a reward signal that tries to measure whether the reasoning steps genuinely cause the answer, then uses that signal during reinforcement learning. The name stands for Causality Reward on Scrambled CoTs.
What problem is CRoS actually solving?
Reinforcement learning with verifiable rewards, RLVR, is the standard recipe now for teaching models to reason. You generate a chain of thought, check if the final answer is correct against a verifier, and reward accordingly. Simple, scalable, and it works well enough that nearly every reasoning model ships on some version of it.
The catch the authors name is that outcome correctness does not imply the CoT contributed to that outcome. A model can produce a right answer with reasoning that is irrelevant, circular, or post-hoc. Reward the outcome alone and you reinforce whatever got you there, including the habit of writing reasoning that looks good but does no work.
This matters beyond academic tidiness. If you are relying on a model’s visible reasoning to audit its behavior, catch errors, or trust it on a hard task, a decorative CoT is worse than no CoT. It gives you confidence you did not earn.
Why does scrambling beat deleting?
People have tried to measure CoT causality before. The usual move is to perturb the chain: delete a reasoning step, inject an error, and see how much the final answer changes. If the answer flips when you break the reasoning, the reasoning must have mattered. If nothing changes, it was decoration.
The authors argue this approach has a confound baked in. Deleting a step or injecting an error does not just break the causal structure, it also removes or corrupts information. So when the answer changes, you cannot tell whether the model was sensitive to the broken reasoning chain or just sensitive to the missing information. High “causality” scores under these methods may be measuring the wrong thing. And when you turn a confounded metric into a training reward, you get bad behavior downstream: the paper reports it pushes models toward less informative chains of thought and less exploration and self-correction, because those are the behaviors that game the flawed signal.
CRoS changes the perturbation. Instead of deleting steps, it scrambles their order. Same content, same information, different sequence. The idea is that genuine reasoning depends on order (step three builds on step two builds on step one), so shuffling it should break a truly causal chain while leaving a decorative one roughly unaffected. Because no information is removed, the confound the earlier methods suffered from mostly goes away. You are testing structure, not content.
That is the clever part. It is a small change in how you poke the chain, and it isolates the thing you actually wanted to measure all along.
What do the numbers show, and how far do they go?
The experiments run on Reasoning Gym with Gemma-3-4B, a small model. Three results stand out, all reported by the authors:
Out-of-domain accuracy: CRoS hits 46.8% on five unseen logic and games datasets, against 43.8% for the strongest baseline. A three-point gain on held-out tasks is the number I care about most, because it suggests the model learned something that transfers rather than memorizing the training distribution.
CoT quality: measured by CoT-Pass@K from K=1 to 128, CRoS averages 43.4% versus 41.4% for baselines, and the paper says it leads consistently across that whole range of K. That consistency matters more than the average. A method that only wins at K=1 might just be luckier on the first try.
Faithfulness: this is the striking one. The hint acknowledgment score jumps to 34.0% from 20.2%. Hint acknowledgment tests whether a model that used a hint actually says so in its reasoning instead of pretending it solved the problem unaided. Nearly doubling that score means the visible reasoning is tracking what the model really did far more often.
Now the honest caveats. This is one model, one 4B-parameter model, on logic and games benchmarks inside Reasoning Gym. It is not GPT-class, not a frontier reasoning model, not math or code at the scale where RLVR gets most of its press. The scrambling trick also carries an assumption worth questioning: that real reasoning is always order-dependent. Plenty of valid reasoning is partly parallel, where several independent observations feed one conclusion and their order does not matter much. For those cases, scrambling might not break a genuinely causal chain, which would make CRoS under-reward good reasoning rather than over-reward bad. The paper does not, from what the sources show, settle how often that happens at scale.
Why should an operator care about faithfulness at all?
Because “the model showed its reasoning” is becoming a load-bearing claim in products, agents, and compliance stories, and most of the time nobody has checked whether the reasoning is real.
If you ship an agent that writes out its plan before acting, you are implicitly telling users and auditors that the plan reflects the decision. If the chain is decorative, your logs are theater. Faithfulness work like CRoS is the quiet infrastructure that makes “show your work” mean something. The faithfulness metric here, hint acknowledgment, is exactly the kind of thing you would want in an eval suite for any system where you plan to trust or audit the reasoning trace.
Practitioner’s take: you cannot run CRoS yourself without a training pipeline, so the usable idea is the measurement, not the reward. Borrow the scramble test for your own evals. Take a sample of your model’s chains of thought, shuffle the order of the steps while keeping every step intact, rerun, and see how often the final answer changes. If it rarely changes, your model is probably decorating, and you should stop treating those traces as explanations. Pair that with a hint acknowledgment check: feed the model an answer hint and count how often it admits using it. Both tests cost you inference calls, not training runs, and they tell you whether the reasoning you are logging is doing any work. The catch most readers will miss: a high scramble-sensitivity score is necessary but not sufficient. A model can be order-sensitive and still wrong. CRoS buys you faithfulness, the reasoning matches the answer, not correctness. Those are different problems, and you still own the second one.