Short answer. AI agents need data that represents actions over time, not prompt-and-response pairs. The training and evaluation unit is a trajectory — a task specification, the tools available, each action taken, the environment's response, and the final outcome — annotated for where the first consequential failure occurred. Alongside it sits a verifier that checks the end state, written before the agent runs.
Key takeaways
- A trajectory, not a single prompt-response pair, is the basic unit of agentic training and evaluation data.
- A verifier is a deterministic check of the task's end state, written before the agent runs, that turns a plausible-looking answer into a repeatable success signal.
- Step-level failure attribution — planning, tool selection, arguments, state tracking, or verification — is what makes a failed run useful rather than just a discarded one.
- Task success rate and valid-path success rate are two different numbers; a gap between them means the agent reached the goal through an unsanctioned route.
- Multilingual or market-specific agents need reviewers working in-market, because acceptable actions and expected phrasing vary by locale.
How is agentic data different from ordinary LLM data?
A conventional language-model example contains one prompt and one desired response; an agent decides, acts, observes the result, updates its plan, and continues until a goal is reached or abandoned. That difference changes what the data has to record, who has to review it, and how success is defined.
| Ordinary LLM data | Agentic data | |
|---|---|---|
| Unit | One prompt and one response | One trajectory across many steps |
| What is judged | The output text | The output and the path taken to it |
| Ground truth | A reference answer or a preference between answers | A verified end state, plus an acceptable-path constraint |
| Failure attribution | The answer was wrong | Planning, tool choice, arguments, state tracking, recovery, or verification — each a separate class |
| Environment | None | Tools, APIs, files, databases, or a simulator, each with its own state |
| Annotator skill | Language and domain judgement | The same, plus the ability to read an execution trace |
| Cost driver | Length and subject difficulty | Trajectory length and the number of decision points needing review |
The row that drives the rest is failure attribution. A single label saying the task was not completed tells you nothing you can act on. Teams that treat agentic data the same way as agentic and generative AI systems are often built and evaluated usually discover the mismatch during evaluation rather than during collection. An agent that chose a sensible plan and called a tool with a malformed argument needs a different fix from one that invented a tool that does not exist, and both differ from one that completed every step correctly and never checked its own work.
What does a trajectory actually contain?
Each step preserves the instruction in force, the tools available at that moment, the action selected, its arguments, the environment's response, the resulting state, and any reasoning the system exposed.
A trajectory is the full, step-by-step record of an agent's run — recorded that way, it reads as an auditable sequence rather than a transcript. A short worked example — an agent asked to reconcile a customer refund against an order record:
| Step | Action taken | Environment response | Annotation |
|---|---|---|---|
| 1 | search_orders(customer_id) |
Three orders returned | Plan valid; tool appropriate |
| 2 | get_order(id=…) on the most recent order |
Order found; status "shipped" | Correct tool, wrong record — the refund request named an earlier order |
| 3 | issue_refund(order_id=…, amount=…) |
Refund created | First consequential failure. Acted on the unverified record; no confirmation step |
| 4 | Reports success to the user | — | Outcome reported as complete; verifier fails on end state |
Two things fall out of that table. The final output looked correct — the agent did issue a refund and did report it fluently — and the error entered at step 2, two steps before anything visibly went wrong. Only step-level annotation locates it. This is also why the first consequential failure is marked explicitly: everything after it is contaminated, and grading those later steps as independent errors inflates the failure count and hides the cause.
What is trajectory annotation?
Trajectory annotation reviews a sequence and marks properties at each step: whether the plan was valid, whether the tool call was appropriate, whether the agent recovered from an error it caused, and where the first consequential failure occurred.
It is valuable for both training and diagnosis, because it records how a result was reached rather than only whether the final output looked right. The labels are only comparable if the failure classes are fixed in advance:
| Failure class | What it looks like in the trace |
|---|---|
| Planning | A coherent goal decomposed into steps that cannot achieve it |
| Tool hallucination | A call to a tool or parameter that does not exist |
| Tool selection | A real tool used for the wrong purpose |
| Argument error | The right tool called with malformed or wrong values |
| State tracking | Acting on stale or misremembered environment state |
| Looping | The same action repeated without new information |
| Premature completion | Task reported complete with criteria unmet |
| Unsafe or unauthorised action | A step outside the permitted scope, whether or not it succeeded |
A fixed taxonomy becomes more useful over time, because it lets a team track which classes are shrinking between model versions and which are persistent — the same discipline behind reasoning trace data that teaches a model to show its work. A class that never shrinks is usually a data problem or an environment problem rather than a model problem.
Why do agents need verifiers and objective success criteria?
Agents usually operate where the final state can be checked — a file should exist, a database record should have changed, a test should pass, a reservation should satisfy stated constraints — and verifiers convert those requirements into repeatable evaluation or reward signals.
A verifier is a deterministic check of the end state, written before the agent runs, because a criterion written afterwards tends to describe what the agent did rather than what it should have done.
Task success rate = Trajectories meeting every stated success criterion ÷ Trajectories attempted
Report it alongside a second figure, because an agent can reach the right end state by an unacceptable route:
Valid-path success rate = Successful trajectories with no unsafe or unauthorised step ÷ Trajectories attempted
The gap between the two numbers is the part that matters for deployment. An agent whose task success rate is high and whose valid-path rate is much lower has learned to achieve goals in ways you have not sanctioned, and averaging the two conceals exactly that. Where the evaluator is only a language model judging another language model, hidden errors pass — the circularity examined in how reliable an LLM is as a judge. Combine deterministic checks with human review for the subjective or high-risk parts; the broader design question of what to check and how is covered in designing enterprise evaluation benchmarks.
How should agent failures be turned into training data?
Keep failed trajectories rather than discarding them, because they carry the most informative material a programme produces.
- Label the reason against the fixed taxonomy, at the step where the failure entered.
- Ask what the failure indicts. An unclear instruction, missing tool documentation, an environment mismatch and a genuine planning weakness all present as a failed run and need different responses.
- Write a corrected demonstration for the same task — a trusted step-by-step example of how it should have been completed, including the correct actions and the final state.
- Build targeted tasks that isolate the weakness, rather than more tasks in general.
- Retest on held-out variants. A fix validated on the task it was written for measures memorisation, not capability.
Corrected demonstrations and preference data between trajectories are also what human-feedback training methods consume — the approach described by Ouyang et al. in "Training language models to follow instructions with human feedback" (arXiv 2203.02155), applied to sequences of actions rather than single answers. Deciding which of these approaches to buy is covered in RLHF, SFT and distillation: what to buy.
When is expert review required?
Whenever the agent's actions touch specialised systems or high-impact decisions — a software agent needs reviewers who can read the code it wrote, a financial workflow needs domain and compliance expertise, an internal operations agent needs someone who knows the organisation's policies well enough to see when a step exceeded them.
The reviewer's job covers both task completion and process quality. An agent that reached the right result through an unsafe or unauthorised path should not be recorded as a success — and a reviewer without the relevant expertise will record it as one, because the end state looks correct.
What should you ask a supplier of agentic data?
Ask what is actually delivered per trajectory, how failures are classified, and who is qualified to write and check verifiers — the answers separate a shallow review process from one that produces usable diagnostic data.
- What exactly is delivered per trajectory — full step-level records, or an outcome label?
- Which failure taxonomy is used, and can it be extended to our environment?
- How is the first consequential failure identified and by whom?
- Are verifiers written before the run, and who writes them?
- How is agreement measured between reviewers on the same trajectory?
- What are the reviewers' qualifications for our domain, and how are they verified?
- Are failed trajectories retained and delivered, or filtered out?
How does Lifewood support agentic AI data programmes?
Lifewood supports the data operations around agent programmes — task creation, step-level trajectory review, failure labelling and human evaluation — using the same structure it applies to annotation generally.
That structure is a fixed taxonomy agreed before work starts, reviewers qualified for the domain being judged, and dual-layer human-in-the-loop review held to a 95%+ accuracy SLA. Where an agent operates in more than one market, the review has to happen in-language, which is what 50+ languages and 40+ delivery centres across 30+ countries support. Related service pages cover enterprise LLM training data and AI data validation.