How to Build Reasoning Trace Data That Teaches Models to Show Their Work

Outcome-only supervision provides little information about which intermediate steps were helpful when the final answer is wrong. In reinforcement learning setups where the verifier only checks the final answer, long reasoning traces produce many samples receiving identical outcome rewards, which weakens the learning signal to the point where the model cannot tell a lucky guess from sound reasoning.

The fix is to label the steps. That sounds obvious and it is operationally difficult, because it requires a kind of annotation most data programmes are not set up to produce — closer to the domain-expert review used for enterprise LLM training data than to routine labelling.

Outcome reward models (ORMs) are reward models that score only a reasoning chain's final answer, providing one bit of signal for the entire chain. They are cheap to label, since you only need to know whether the result was right.

Process reward models (PRMs) are reward models that score the correctness of each intermediate reasoning step rather than just the final answer. They have been used for reranking, search and test-time scaling by evaluating the quality of the reasoning process itself. Step-level annotation is more expensive per example than outcome-level annotation but produces substantially stronger supervision signals for reasoning quality.

There is also a subtler benefit. A model trained only on outcomes can learn to reach right answers through reasoning that does not hold up, because nothing in the training signal penalised the bad path. Process supervision is what makes the visible reasoning trustworthy rather than decorative, which is why it increasingly sits alongside RLHF, SFT and other post-training data that enterprise teams buy.

First-error identification is the finding, established by Lightman et al. (2023), that labelling only the first incorrect step in a chain is sufficient supervision for effective PRM training. Once the first error is found, all preceding steps are annotated as correct and all subsequent steps as incorrect.

That converts an exhaustive labelling task into a search problem, and search can be optimised. OmegaPRM (Luo et al., 2024) employs binary search to locate the first erroneous step, reducing the number of evaluations required and, as the authors describe it, emulating the human annotation process without labelling every step. For a twelve-step trace, binary search means roughly four judgements rather than twelve — at scale, the difference between a viable programme and an unaffordable one.

Annotating reasoning traces requires a fundamentally different annotator profile than standard labelling tasks. A mathematics trace needs someone who can spot that step seven applied a valid operation to the wrong quantity. A code trace needs someone who can see that the logic is sound but the loop boundary is off by one. A legal or medical reasoning trace needs someone qualified in that domain — the kind of vetting described in how annotators are recruited, trained and certified for specialist domains.

This is expert review, and it should be staffed and priced accordingly. A team recruited and trained for bounding boxes or sentiment labels cannot do it, and putting them on the task produces a dataset that looks complete and teaches the model nothing useful.

The label set used in practice distinguishes four states rather than a binary: correct and necessary steps that advance the reasoning; correct but redundant steps, which matter because redundancy is a real quality problem in generated reasoning; incorrect steps; and incomplete steps requiring additional information to validate.

Each step receives an independent label applied by a qualified reviewer evaluating that step in isolation from the final answer. A reviewer who knows the final answer was correct will unconsciously rationalise a flawed intermediate step, and a reviewer who knows it was wrong will hunt for errors that are not there. Hiding the outcome during step review is what keeps the labels honest.

Two quality dimensions matter, and they pull in different directions. Accuracy and logical coherence at each step are the critical floor. Diversity of reasoning paths matters for generalisation: training on a narrow set of reasoning patterns produces a model that is brittle on problems requiring a different approach, so a thousand traces that all solve problems the same way is worse than five hundred that solve them five different ways.

The tension is that the easiest way to get high-quality traces is to generate them from a strong model with a fixed prompt template, which produces exactly the narrow, homogeneous reasoning that limits generalisation. Diversity has to be engineered deliberately, not hoped for.

Inter-annotator agreement is the measured rate at which independent qualified reviewers assign the same correctness label to a reasoning step. High agreement on step correctness, measured on a calibration set before full annotation begins, is the signal that the programme is working — the same discipline covered in gold sets, audit sampling and consensus for QA on annotated data generally. If two qualified experts disagree on whether step seven is correct, either the step is genuinely ambiguous or the guideline for what counts as a step is unclear, and both need resolving before scale.

Relying on humans to label data is an important bottleneck in scaling process-level datasets; annotating fine-grained step rewards typically requires human experts or high-performing models, which is labour-intensive and costly either way. Several automated approaches exist. Rollout-based estimation (MathShepherd) generates completions from partial reasoning chains and measures the percentage that reach the correct solution, using that as the step's value — effective, but computationally heavy. Binary search over rollouts (OmegaPRM) combines rollout evaluation with binary search to cut cost. Information-theoretic labelling uses Monte Carlo Net Information Gain to derive step labels from information gain. Contrastive approaches such as CPMI quantify changes in predicted probability between correct and incorrect answers at each step, curated into the CPMI-80k dataset. ProcessThinker, published at ICLR 2026, avoids a separate reward model entirely by rewriting traces into a step-tagged format for cold-start fine-tuning, then applying reinforcement learning with a rollout-based process reward.

The caveats are consistently reported. Automated approaches relying on Monte Carlo rollouts or search can be noisy and sensitive to how "steps" are defined — a design decision, not a property of the data. Step entropy research also shows not all steps contribute equally to predictive power, so uniform labelling effort is inefficient. The pragmatic architecture is automation for coverage and volume, human expert review for the calibration set, the ambiguous cases, and the domains where a wrong step has real consequences — the same judgment call at issue in how reliable an LLM is as a judge for automated evaluation more broadly.

Most reasoning trace work is done in English and mathematics, for understandable reasons: verifiable answers, abundant source material, available expertise. Research on cross-lingual generalisation found that reinforcement learning on Chinese reasoning data improved performance not only in Chinese but substantially on German, Spanish and Bengali evaluations, well beyond what supervised fine-tuning on the same data achieved. Reasoning skill, as distinct from factual recall, appears to transfer between languages remarkably well.

That changes the investment case: process supervision in one language buys more elsewhere than knowledge data does, in contrast to most multilingual data collection work, where factual knowledge in one language does not transfer to another. But it does not buy everything — a clinical reasoning trace referencing one country's drug names, or a financial one referencing one country's tax treatment, still needs building per market regardless of how well the underlying reasoning skill transfers.

Declaring the interest: this work sits alongside RLHF and preference data, with review across 50+ languages. Two observations from that vantage are worth stating independent of who does the work.

The first is a staffing observation. Reasoning trace annotation breaks the standard annotation operating model: you cannot recruit a general pool, train them on guidelines and scale. It needs domain-qualified reviewers, which means a different recruitment channel, a different pay band, a smaller available pool per domain, and a longer ramp. Programmes that budget for reasoning traces at ordinary annotation rates discover this several weeks in, and the usual outcome is either a quality collapse or a renegotiation.

The second is about calibration. The measurement that matters is inter-annotator agreement on step correctness, established on a calibration set before scaling. In practice the first calibration round on a reasoning task tends to produce lower agreement than teams expect, and the reason is almost always segmentation: two experts disagreeing about whether step seven is correct are frequently disagreeing about where step seven begins. Fixing the step definition resolves more disagreement than retraining the annotators does — the domain is harder and the reviewers are more expensive, but the discipline is the same one behind enterprise LLM training data generally.

From there: use first-error identification rather than exhaustive labelling, since Lightman and colleagues established that this is sufficient for effective PRM training and binary search makes it efficient. Recruit domain experts, not general annotators, and budget for a longer ramp — the kind of vetting laid out in writing annotation guidelines that annotators actually follow applies here with a much higher bar. Judge steps in isolation from the final answer, since that is the control that keeps labels honest. Run a calibration set and measure step-level agreement before scaling, and treat low agreement as a segmentation problem first and a training problem second. Engineer path diversity deliberately, since generating traces from one model with one template produces homogeneous reasoning and a brittle result. Use automation for coverage and humans for the hard cases, rather than choosing between them. And do not assume outcome correctness implies process correctness — a trace that reaches the right answer through flawed reasoning is a negative training example wearing a positive label, and it is the most damaging record type in the dataset. Teams comparing vendors for this kind of work can start from a comparison of data annotation companies for LLM training and generative AI.