Short answer. Production logs become training data through a data flywheel: capture interactions, scrub them of personal data, triage failures automatically, confirm and label the genuine ones with human experts, curate a small set, fine-tune, evaluate and redeploy. NVIDIA's published agent flywheel shows the proportions: 495 unsatisfactory responses, 140 flagged by an LLM judge, 32 confirmed by experts, and a curated set of 685 data points that measurably improved smaller models. The judge filters; humans decide.
Key takeaways
- NVIDIA's internal agent flywheel took 495 unsatisfactory production responses, an LLM-as-a-judge flagged 140 as routing failures, and subject matter experts confirmed only 32, roughly a 77% false positive rate on automated triage.
- Raw production logs are not training data: they are noisy, biased toward easy cases, full of personal data, and only become useful after scrubbing, triage, root-cause sorting and human curation.
- Cycle time from log to redeployment is the binding constraint on a data flywheel, not dataset size; NVIDIA's curated set was 685 data points and practitioners recommend starting with 100 to 200 labelled examples.
- A flywheel amplifies the languages, markets and segments a product already serves and cannot bootstrap the ones it does not; those need commissioned data collection until usage exists.
- Training on model outputs judged only by models degrades performance, with rare cases disappearing first; human review is the external anchor that stops the loop consuming its own errors.
What does NVIDIA's flywheel show about turning logs into training data?
NVIDIA's published account of its internal agent flywheel shows that automated evaluation narrows the review population and human experts make the actual judgement, and that a few hundred curated examples were enough to improve smaller models. The numbers tell the story better than any diagram.
The team took 495 unsatisfactory responses from production. An LLM-as-a-judge pass identified 140 as caused by incorrect routing. Subject matter experts then reviewed those 140 by hand and confirmed 32.
One hundred and forty flagged. Thirty-two real. That is roughly a 77% false positive rate on automated triage, in a well-resourced programme run by people who build this infrastructure for a living, and it is not a failure of the approach. It is what the approach is supposed to do: automated evaluation reduces 495 to 140 so that expensive human review runs on 140 rather than 495. How far to trust that filter is covered in how reliable an LLM is as a judge.
The final output was a curated ground truth dataset of 685 data points, split 60/40 for training and testing, and NVIDIA reported significant improvements in smaller-model performance from it. Six hundred and eighty-five, not six hundred and eighty-five thousand.
What is a data flywheel, and why do static datasets fail in production?
A data flywheel is a feedback loop in which data collected from live interactions is used to continuously refine an AI model, which produces better outcomes and more valuable data. It replaces the older pattern of collecting a fixed corpus, annotating it once, training and deploying.
The linear approach fails quickly in production, where a model meets prompts, domain shifts and long-tail edge cases no pre-training corpus captured, and a dataset frozen at launch cannot follow them. The flywheel's premise is self-reinforcing: as more users interact with the system, the logs capture real successes, user corrections and agent failures; extracting, annotating and training on those failures improves the next iteration, which drives more usage and richer logs.
The strategic claim underneath is where the business case sits. The thesis in NVIDIA's and Iguazio's reference material is that continuous improvement of AI agents does not require constantly upgrading to larger models. It requires systematic feedback loops that let smaller, cheaper, faster models match or approach the accuracy of larger ones. That reframes the flywheel from a quality initiative into a cost programme.
One correction to the framing: raw production logs are not training data. They are unstructured, noisy, weighted toward routine queries that carry no new learning signal, and laden with compliance risk. Log volume does not spin the wheel faster; the speed of a flywheel is set by how efficiently failures are found, classified and annotated, and feeding uncurated logs straight back into fine-tuning dilutes accuracy and amplifies drift.
How does the loop run, stage by stage?
The loop runs capture, monitor, automated triage, human review, curation, customisation, evaluation and redeployment, with an orchestrator deciding when each stage fires. Each stage has a distinct input and output, and conflating them is the usual reason flywheel investments stall.
| Stage | What happens | Output |
|---|---|---|
| Capture | Inference logs, prompts, responses, retrieval traces, tool calls, latency and user feedback, stored somewhere queryable (Elasticsearch in the NVIDIA and MLRun reference architecture) | Raw telemetry stream |
| Scrub | PII detection and de-identification; residency and consent rules applied before anything reaches training storage | Compliance-cleared telemetry |
| Monitor and detect | Performance, stability and resource usage tracked continuously, surfacing candidates rather than waiting for complaints | Candidate pool |
| Automated triage | LLM-as-a-judge and online evaluation narrow the pool; the 495 to 140 step | Flagged failures |
| Root-cause sorting | Each flagged failure mapped to a collection, annotation or modelling problem | Remediation queue |
| Human review | Domain experts confirm which flags are genuine and define correct behaviour; the 140 to 32 step | Ground truth labels |
| Curate | Confirmed cases become a training and evaluation set, split for both | Gold SFT, preference or process-supervision data |
| Customise | Fine-tuning through LoRA, p-tuning or supervised fine-tuning against the curated set | Candidate model |
| Evaluate | Candidate models run against production logs and held-out data using zero-shot, RAG and LLM-as-judge evaluation, plus regression tests and red-teaming | Go or no-go decision |
| Redeploy | The improved model ships and generates the next round of logs | Next cycle |
Orchestration turns this from a project into a system: in the reference implementations an orchestrator wraps the loop, triggering evaluation and customisation when logs meet defined conditions and escalating to humans where a decision needs one. The same capture, review and retrain pattern appears in the published Agent-in-the-Loop flywheel for LLM-based customer support.
How do you scrub production logs before they touch training?
Production logs are user data and must be de-identified, residency-checked and consent-checked at capture, before they are stored for training. Retrofitting a lawful basis across a year of accumulated logs is far harder than writing it correctly on day one.
Logs routinely contain personally identifiable information, proprietary customer payloads and sometimes operational credentials. Regex matching alone is not sufficient for generative, multimodal applications; enterprise pipelines use multi-pass de-identification combining named-entity recognition with tokenisation or differential-privacy techniques, and keep the output as a separate compliance-cleared stream.
Residency is the part almost nobody addresses. A flywheel that captures logs in one jurisdiction and processes them for training in another is a cross-border data transfer, whether or not anyone in the pipeline calls it one. Consent language written for service delivery does not automatically cover model training. Lifewood, as one example of how a managed provider handles this, states that it enforces governance at intake across its 40+ delivery centres, including EU-only processing through geographic access controls (company-reported).
How do you triage failures and find the root cause?
Sample for high-entropy interactions rather than uniformly, then sort every confirmed failure into one of three buckets: missing data, bad labels or guidelines, or a model capacity limit. Only the first two are data problems, and only they belong in the flywheel's annotation queue.
The signals that mark a likely failure are explicit user edits or rejections, low confidence scores, abandoned sessions, execution timeouts and downstream task exceptions. Clustering by embedding distance or step entropy groups them into stratified failure clusters and a long-tail queue.
Once a failure is isolated, the root-cause taxonomy used in collecting training data for generative AI applies:
- Absent data (a collection problem): the user prompted in a low-resource dialect or domain terminology missing from the training corpus. Remedy: commission data.
- Mislabelled data (an annotation problem): conflicting guidelines in earlier fine-tuning or RLHF runs produced erratic behaviour. Remedy: fix the guidelines and relabel.
- Capacity limits (a modelling problem): the model lacks the context window or architectural depth to solve the task. Remedy: an engineering change, not more labels.
Conflating the three is how budgets get spent retraining models for problems that only needed curation, or labelling data for problems no data will fix.
Why is cycle time the real constraint?
Cycle time from log to redeployment, not dataset size, is the measure that decides whether a flywheel works, because a loop slower than the production distribution is training on a world that no longer exists. A loop that closes in days against 200 examples beats one that closes in a quarter against 20,000.
Arize makes the argument most directly, and it explains why manual flywheels fail rather than merely underperform. Manual processes break down at scale. By the time examples have been reviewed, labelled, retrained and deployed, the production system has moved on: requirements changed, new edge cases appeared, and the team is always catching up.
That is a different failure mode from not having enough data. A flywheel that takes eleven weeks to complete a cycle is not slow; it is stationary, producing improvements calibrated to a distribution that has already shifted.
Where do flywheels quietly go wrong?
Four failure modes recur, none of which appear in vendor diagrams: survivorship bias in the logs, automated triage treated as ground truth, training on unverified model outputs, and privacy or residency handled as an afterthought.
Survivorship bias in the logs
Production logs contain interactions from users who stayed. The user who asked something the system handled badly and never came back generates one bad log and then nothing; the user who found it useful generates hundreds. So the log distribution over-represents what already works, and the flywheel optimises hardest for the cases that needed the least help. This is structural and does not fix itself with volume. It needs counterweighting: sampling abandoned sessions, tracking first-interaction churn, and treating the absence of logs from a segment as a finding. The techniques in long-tail and edge-case mining are the counterweight.
Automated triage treated as ground truth
The 140 to 32 funnel is the warning. Had NVIDIA fine-tuned on all 140 flagged cases, roughly three quarters of that training signal would have been wrong, and the model would have learned to correct routing decisions that were already correct. Automated evaluation narrows the search space. It does not make the judgement.
Training on your own outputs without verification
A flywheel that captures model responses, judges them with a model and trains on the result is a recursive loop with no external anchor. The same applies to the tempting shortcut of auto-labelling high-confidence outputs and appending them to the SFT set. The model collapse literature is clear that recursive training on generated data degrades models, with rare cases disappearing first, and that the mitigation is verification and accumulation of real, human-anchored data rather than replacement. The evidence is set out in whether it is safe to train on AI-generated data. Human review is not a quality nicety in this loop; it is the anchor that stops it eating itself.
Privacy and residency as an afterthought
A pipeline that discovers a year in that its logs are personal data must either delete them or retrofit consent. Scrubbing and residency belong at the capture stage.
What can a flywheel not do?
A flywheel amplifies what a product already has and cannot bootstrap what it does not. No users in a market means no logs from that market, no improvement there, and continued low adoption, so entering new languages, domains or segments requires commissioned collection rather than harvested traffic.
If a product has no users in Indonesia, its logs contain no Indonesian interactions. The flywheel cannot improve Indonesian performance because there is nothing to feed it, and since the model performs poorly in Indonesian, adoption stays low and the logs stay empty. The loop runs in reverse: weak coverage produces weak usage produces no data. The same applies to any segment served badly enough that users leave, any domain not yet entered, and any edge case rare enough not to appear. Those require commissioned collection: data produced deliberately, by native speakers recruited in the target market, rather than harvested from traffic that does not exist.
This is where Lifewood's own work sits, so the interest is declared. Lifewood collects and annotates data across 50+ languages through managed multilingual data collection, and the recurring pattern is a client with a mature flywheel in two or three languages and flat performance in the markets they want to grow into. The flywheel is working as designed; it cannot manufacture logs from users who are not there yet. One company-reported example: a foundation-model corpus programme delivered 2.1 billion tokens across 42 languages at a 97.3% quality acceptance rate, and adding 18 low-resource languages to the training mix for the first time was followed by a 40% reduction in downstream toxicity benchmarks.
The practical framing: flywheel for the languages you have, commissioned collection for the languages you want. Once a market reaches enough usage the flywheel takes over and the collection cost stops. The providers that do this work at scale are compared in the top multilingual AI data collection companies.
Where does human expertise fit, and why was 685 examples enough?
The value of NVIDIA's 685-point dataset was concentrated entirely in curation: the automated pass found candidates and subject matter experts decided which were real and what the correct behaviour should have been. A small, correct, well-targeted dataset outperforms a large, noisy one, and the cost sits in making it correct rather than making it large.
An automated flywheel does not eliminate human annotators; it raises the bar for them. Automated systems filter volume; humans resolve ambiguous cases, evaluate reasoning and define what correct looks like. Three roles are load-bearing:
- Subject matter experts who can judge whether a flagged failure is genuinely a failure. This requires domain knowledge, not annotation training.
- Reviewers who define correct behaviour, not just identify wrong behaviour. Knowing the routing was wrong is half the label; knowing where it should have gone is the other, harder half.
- An owner of the sampling strategy, because what gets reviewed determines what gets fixed, and automated flagging inherits whatever biases the judge model has.
None of these are jobs a general annotation pool does well. As the data gets closer to judgement, the annotator profile shifts from trained to qualified, the same conclusion reached for reasoning traces and preference data, and the reason enterprise LLM training data programmes are staffed differently from bulk labelling. Three situations make the requirement concrete.
Language and dialect precision
When an assistant receives logs from non-English markets, generic crowd annotators frequently miss regional idioms and low-resource dialects, so localised failures get mislabelled and the flywheel learns the wrong correction. Recruiting native-speaker, in-community annotators from the language communities where the logs originate keeps the curated set culturally valid.
Step-level reasoning failures
When logs reveal multi-step reasoning failures in code generation, medical analysis or mathematical logic, marking the final answer wrong is insufficient. The flywheel must capture process-level data: an expert isolates the precise intermediate step where the logic broke and labels it. Lightman et al. showed that this kind of process supervision outperforms outcome-only supervision, and the OmegaPRM work showed that a binary search for the first erroneous step cuts annotation cost while still providing a strong signal for process reward models. The mechanics are covered in building reasoning trace data.
Multimodal logs
Flywheels are not limited to text. In autonomous driving and computer vision, edge-case logs include sensor calibration drift, camera occlusions and unusual traffic environments; curating them means 3D bounding boxes, LiDAR point-cloud segmentation and sensor-fusion alignment by specialised annotators, with a second independent review pass against gold calibration sets before release.
How do you know the flywheel is working?
A working flywheel shows a steady decline in known error classes after each cycle, high agreement among the experts labelling its calibration sets, and better benchmark gains per token than uncurated data would give. Cycle time is the operational metric that sits above all three.
- Failure reduction rate per iteration: the percentage drop in recurring long-tail failure modes after a fine-tuning cycle. A healthy flywheel retires error classes; an unhealthy one keeps finding the same ones.
- Inter-annotator agreement on calibration sets: consistency among the expert reviewers handling triaged logs. Run a calibration batch and measure agreement before releasing a dataset to training, so the model is not trained on conflicting guidelines. Lifewood's managed programmes use a 95%+ inter-annotator agreement threshold measured against a customer-approved gold set.
- Data efficiency: performance gain per unit of curated data, with the NVIDIA case as the benchmark for what a small, failure-curated set should achieve.
- Cycle time from log to redeploy, the primary operational metric, because every other number only matters if the loop closes faster than the distribution moves.
How do you get started without building the full stack?
Start with one improvement area, one focused evaluation, 100 to 200 labelled examples and a mandatory human confirmation step, and measure cycle time from the first iteration.
- Pick one high-value improvement area: safety, accuracy or response style. Not all three.
- Write ingestion specifications first: metadata schema, PII scrubbing rules, residency and consent framework. Sort this out at capture, not later.
- Set up online evaluation for that specific concern rather than general quality monitoring, with triage filters built on user edits, confidence thresholds and timeouts.
- Categorise root causes before annotating, so engineering effort goes to modelling problems and annotation effort goes to data problems.
- Collect 100 to 200 labelled examples, the recommended starting volume; the NVIDIA case suggests the ceiling for useful results is lower than most teams assume.
- Put a human confirmation step between automated flagging and dataset inclusion, always. The 140 to 32 ratio is the argument.
- Run a calibration batch and measure inter-annotator agreement before releasing any dataset to training.
- Instrument for what is missing, not just what failed: abandoned sessions, unanswered queries, segments generating no traffic.
- Measure cycle time from log to redeploy as the primary operational metric.
Where the review step needs domain experts, native speakers or specialised annotators the team does not have, a managed delivery partner with regional language coverage and enforceable accuracy SLAs is the usual answer; an unvetted crowd reintroduces the noise the flywheel exists to remove.