Short answer. Outcome supervision tells a model its answer was wrong but not which step broke, so long traces collect many correct steps under one negative signal. Process reward models fix that by scoring intermediate steps rather than final answers, and they need step-level annotation — more expensive per example, substantially stronger supervision. The cost is manageable because Lightman et al. (2023) showed effective PRM training only requires identifying the first erroneous step; everything before it is correct by construction. OmegaPRM then finds that step by binary search rather than labelling every one.
That Teaches Models to Show Their Work?
Here is the problem in one sentence, and it is the reason this entire field exists.
When a model works through a twelve-step problem and gets the wrong answer, outcome supervision tells it the answer was wrong. It says nothing about which of the twelve steps broke.
The research literature puts it more precisely: outcome-only supervision provides little information about which intermediate steps were helpful when the final answer is wrong. And in reinforcement learning setups where the verifier only checks the final answer, long reasoning traces produce many samples receiving identical outcome rewards, which weakens the learning signal to the point where the model cannot tell a lucky guess from sound reasoning.
The fix is to label the steps. That sounds obvious and it is operationally difficult, because it requires a kind of annotation most data programmes are not set up to produce.
Outcome versus process, and why the distinction is the whole thing
Two families of reward model sit behind this.
Outcome reward models (ORMs) evaluate the final answer. Cheap to label, since you only need to know whether the result was right. The supervision signal is one bit for an entire reasoning chain.
Process reward models (PRMs) assess the correctness of intermediate reasoning steps rather than relying solely on final answers. They have been used for reranking, search and test-time scaling by evaluating the quality of the reasoning process itself.
The trade is straightforward and worth stating plainly: step-level annotation is more expensive per example than outcome-level annotation but produces substantially stronger supervision signals for reasoning quality.
There is also a subtler benefit that gets less attention. A model trained only on outcomes can learn to reach right answers through reasoning that does not hold up, because nothing in the training signal penalised the bad path. Process supervision is what makes the visible reasoning trustworthy rather than decorative.
The finding that makes this affordable
If you label every step in every trace, the cost is prohibitive. There is a result from Lightman and colleagues in 2023 that changes the economics considerably, and it is the single most useful thing to know when scoping this work.
Effective PRM training only requires identification of the first erroneous step in a complete chain-of-thought reasoning chain. Once the first error is found, all preceding steps are annotated as correct and all subsequent steps as incorrect.
That converts an exhaustive labelling task into a search problem. And search can be optimised: OmegaPRM (Luo et al., 2024) employs binary search to locate the first erroneous step, reducing the number of evaluations required and, as the authors describe it, emulating the human annotation process without labelling every step.
For a twelve-step trace, binary search means roughly four judgements rather than twelve. At scale that is the difference between a viable programme and an unaffordable one.
The annotator profile is genuinely different
This is the part most relevant to anyone commissioning this work, and it is where reasoning trace programmes most often fail.
Annotating reasoning traces requires a fundamentally different annotator profile than standard labelling tasks. The requirement is explicit: annotators need domain expertise sufficient to verify that each reasoning step is correct, not just that the final answer matches a known output.
Consider what that means concretely. A mathematics trace needs someone who can spot that step seven applied a valid operation to the wrong quantity. A code trace needs someone who can see that the logic is sound but the loop boundary is off by one. A legal or medical reasoning trace needs someone qualified in that domain.
This is not annotation in the sense that most annotation programmes use the word. It is expert review, and it should be staffed and priced accordingly. A team recruited and trained for bounding boxes or sentiment labels cannot do it, and putting them on the task produces a dataset that looks complete and teaches the model nothing useful.
The label set used in practice distinguishes four states rather than a binary:
Correct and necessary steps that advance the reasoning. Correct but redundant steps, which matter because redundancy is a real quality problem in generated reasoning. Incorrect steps. Incomplete steps requiring additional information to validate.
And the methodological requirement that makes it reliable: each step receives an independent label applied by a qualified reviewer evaluating that step in isolation from the final answer.
That isolation is the discipline. A reviewer who knows the final answer was correct will unconsciously rationalise a flawed intermediate step, and a reviewer who knows it was wrong will hunt for errors that are not there. Hiding the outcome during step review is what keeps the labels honest.
Quality beats volume, and the reason is unusually direct
In most data work, "quality over quantity" is advice. Here it is a mechanism.
A large dataset of low-quality or logically flawed reasoning traces trains the model to reproduce those flaws.
The model is not learning to reach answers. It is learning to imitate reasoning, and it will imitate whatever reasoning you give it, including the bad reasoning.
Two quality dimensions matter, and they pull in different directions.
Accuracy and logical coherence at each step are the critical quality requirements. This is the floor.
Diversity of reasoning paths matters for generalisation. Training on a narrow set of reasoning patterns produces a model that is brittle on problems requiring a different approach. So a dataset of a thousand traces that all solve problems the same way is worse than five hundred that solve them five different ways.
The tension is that the easiest way to get high-quality traces is to generate them from a strong model with a fixed prompt template, which produces exactly the narrow, homogeneous reasoning that limits generalisation. Diversity has to be engineered deliberately, not hoped for.
The measurement that tells you the programme is working: high inter-annotator agreement on step correctness labels, measured on a calibration set before full annotation begins. This is the same discipline as any annotation programme, applied to a harder judgement. If two qualified experts disagree on whether step seven is correct, either the step is genuinely ambiguous or the guideline for what counts as a step is unclear. Both need resolving before scale.
Can this be automated? Partly, and with real caveats
Human labelling is the bottleneck. As one 2026 paper puts it, relying on humans to label data is an important bottleneck in scaling process-level datasets, and annotating fine-grained step rewards typically requires human experts or high-performing models, which is labour-intensive and costly either way.
Several automated approaches exist and it is worth knowing what each actually does.
Rollout-based estimation. MathShepherd generates completions starting from partial reasoning chains and measures the percentage that reach the correct solution, using that as the step's value. Effective, but it requires substantial computation.
Binary search over rollouts. OmegaPRM combines rollout evaluation with binary search to find the first incorrect step, cutting cost substantially.
Information-theoretic labelling. A 2026 framework defines Monte Carlo Net Information Gain to construct step-level supervision, generating structured reasoning outputs, validating final answers with task-specific functions, and deriving step labels from information gain.
Contrastive approaches. CPMI quantifies contrastive changes in the predicted probability of correct versus incorrect answers at each step, which the authors argue eliminates the need for laborious human annotation and costly Monte Carlo rollouts. They curated CPMI-80k, an 80,000-example step-level supervision dataset derived from Math-Shepherd.
Avoiding the separate reward model entirely. ProcessThinker, published at ICLR 2026, rewrites reasoning traces into a step-tagged format for cold-start fine-tuning, then applies GRPO with a rollout-based process reward, sampling multiple continuations from each intermediate step and using the empirical success rate as the step reward. The motivation is that training and maintaining a separate PRM adds engineering overhead and can introduce a mismatch between the PRM and the final policy.
Now the caveats, which are consistently reported and rarely emphasised.
Automated approaches relying on Monte Carlo rollouts or MCTS-style search can be noisy and sensitive to how "steps" are defined. That second point is the one to sit with: the definition of a step is a design decision, not a property of the data, and automated methods inherit whatever definition your segmentation produced.
There is also a finding worth knowing about what steps are actually worth labelling. Work on step entropy as a measure of redundancy and compressibility in reasoning traces shows that not all steps contribute equally to predictive power.
Uniform labelling effort across all steps is therefore inefficient, and the interesting question is which steps carry the signal.
The pragmatic position that emerges: automation for coverage and volume, human expert review for the calibration set, the ambiguous cases, and the domains where a wrong step has real consequences. Which is the same architecture that governs any serious data programme.
The multilingual dimension, and a genuinely encouraging finding
Most reasoning trace work is done in English and mathematics, for understandable reasons: verifiable answers, abundant source material, available expertise.
There is a result worth knowing here that runs the opposite way to most multilingual findings in this series. Research on cross-lingual generalisation found that reinforcement learning on Chinese reasoning data improved performance not only in Chinese but substantially on German, Spanish and Bengali evaluations, well beyond what supervised fine-tuning on the same data achieved. Related work suggests that reasoning skill, as distinct from factual recall, transfers between languages remarkably well.
That is genuinely good news and it changes the investment case. Factual knowledge in one language does not give you factual knowledge in another. But teaching a model to reason carefully appears to be a more portable skill.
Two practical implications follow.
Reasoning trace investment has better cross-lingual return than most data investment. If your budget forces a choice, process supervision in one language buys more elsewhere than knowledge data does.
But it does not buy everything. Domain reasoning that depends on local context, regulation, units, conventions or terminology still needs building per market. A clinical reasoning trace referencing one country's drug names or a financial one referencing one country's tax treatment does not transfer regardless of how well the underlying reasoning skill does.
Where this connects to our own work
Declaring the interest: Lifewood does this work as part of its AI data services, alongside RLHF and preference data, with human-in-the-loop review across 50-plus languages.
Two observations from that vantage that I think are useful independent of who does the work.
The first is a staffing observation. Reasoning trace annotation breaks the standard annotation operating model. You cannot recruit a general pool, train them on guidelines and scale. You need domain-qualified reviewers, which means a different recruitment channel, a different pay band, a much smaller available pool per domain, and a longer ramp.
Programmes that budget for reasoning traces at annotation rates discover this several weeks in, and the usual outcome is either a quality collapse or a renegotiation.
The second is about calibration. The measurement that matters is inter-annotator agreement on step correctness, established on a calibration set before scaling. In our experience the first calibration round on a reasoning task produces lower agreement than teams expect, and the reason is almost always segmentation: two experts disagreeing about whether step seven is correct are frequently disagreeing about where step seven begins. Fixing the step definition resolves more disagreement than retraining the annotators does.
That is the same finding as in any annotation programme, which is oddly reassuring. The domain is harder, the reviewers are more expensive, and the discipline is identical.
What to do if you are scoping this
Decide what a step is, and write it down with examples. This single decision determines annotation consistency, automated labelling reliability and the comparability of your results.
Use first-error identification rather than exhaustive labelling. Lightman and colleagues established that this is sufficient for effective PRM training, and binary search makes it efficient.
Recruit domain experts, not annotators. And budget accordingly, including a longer ramp.
Judge steps in isolation from the final answer. This is the control that keeps labels honest.
Run a calibration set and measure step-level agreement before scaling. Treat low agreement as a segmentation problem first and a training problem second.
Engineer path diversity deliberately. Generating traces from one model with one template produces homogeneous reasoning and a brittle result.
Use automation for coverage and humans for the hard cases, rather than choosing between them.
Do not assume outcome correctness implies process correctness. A trace that reaches the right answer through flawed reasoning is a negative training example wearing a positive label, and it is the most damaging record type in the dataset.
Key takeaways
- Outcome supervision tells a model the answer was wrong but not which step broke. In outcome-only reinforcement learning, long traces produce many samples with identical rewards, weakening the learning signal.
- Outcome reward models evaluate final answers; process reward models assess intermediate steps and support reranking, search and test-time scaling.
- Step-level annotation is more expensive per example than outcome-level but produces substantially stronger supervision for reasoning quality.
- Lightman et al. (2023) established that effective PRM training only requires identifying the first erroneous step: preceding steps are then labelled correct and subsequent ones incorrect.
- OmegaPRM applies binary search to locate the first error, reducing evaluation cost and emulating human annotation without labelling every step.
- Reasoning trace annotation requires a fundamentally different annotator profile, with domain expertise sufficient to verify each step rather than check the final answer.
- The practical label set has four states: correct and necessary, correct but redundant, incorrect, and incomplete pending further information.
- Each step should be judged independently and in isolation from the final answer, since knowing the outcome biases step-level review in both directions.
- A large dataset of logically flawed traces trains the model to reproduce those flaws. The model imitates reasoning, including bad reasoning.
- Diversity of reasoning paths matters for generalisation; narrow pattern coverage produces brittle models, and diversity must be engineered rather than assumed.
- High inter-annotator agreement on step correctness, measured on a calibration set before full annotation, is the signal that the process is producing reliable supervision.
- Automated approaches include MathShepherd rollouts, OmegaPRM binary search, Monte Carlo Net Information Gain, CPMI contrastive labelling with its CPMI-80k dataset, and ProcessThinker's PRM-free approach at ICLR 2026.
- Automated methods relying on rollouts or MCTS can be noisy and are sensitive to how steps are defined, and maintaining a separate PRM introduces engineering overhead and possible policy mismatch.
- Step entropy research shows not all steps contribute equally to predictive power, so uniform labelling effort is inefficient.
- Reasoning transfers across languages better than knowledge does: RL on Chinese reasoning data produced substantial gains on German, Spanish and Bengali evaluations.
- Domain reasoning depending on local regulation, units, conventions or terminology still requires per-market work regardless of that transfer.
Sources and further reading
- Digital Divide Data, "Chain-of-Thought Annotation: How Reasoning Traces Improve LLM Performance", on the annotator profile requirement, the four-state label set, isolation from the final answer, quality over volume, path diversity and calibration-set inter-annotator agreement
- "An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning", arXiv, on Lightman et al. (2023) first-erroneous-step finding and OmegaPRM binary search
- "ProcessThinker: Enhancing Step-Level Process Rewards", ICLR 2026, on sparse outcome supervision, identical rewards across long traces, PRM engineering overhead and policy mismatch, and the step-tagged cold-start approach
- "Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain", arXiv, on human labelling as a scaling bottleneck, MathShepherd rollouts, OmegaPRM cost reduction and step entropy findings
- "Efficient Process Reward Modeling via Contrastive Mutual Information", arXiv, on ORM versus PRM framing, annotation cost, the CPMI approach and the CPMI-80k dataset derived from Math-Shepherd
- "Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs", arXiv, on reasoning transfer from Chinese training data to German, Spanish and Bengali evaluations
- Lifewood, AI data services including RLHF and human-in-the-loop quality assurance