Short answer. NVIDIA's internal agent flywheel is the clearest worked example: 495 unsatisfactory responses, of which an LLM-as-a-judge flagged 140 as routing failures, which subject-matter experts then reviewed by hand. The curated set was 685 data points split 60/40 train and test, and it produced significant gains in smaller-model performance. The important detail is what the judge was for — automated evaluation is a filter that reduces the review population, not a verdict, and treating its output as ground truth is how a flywheel starts training on its own errors.
Training Data?
There is a set of numbers in NVIDIA's published account of their internal agent flywheel that tells you more about how this works than any diagram.
They took 495 unsatisfactory responses from production. An automated LLM-as-a-judge pass identified 140 as caused by incorrect routing. Then subject matter experts reviewed those 140 manually and confirmed 32 cases of genuinely incorrect routing.
One hundred and forty flagged. Thirty-two real.
That is roughly a 77% false positive rate on the automated triage step, in a well-resourced programme run by people who build this infrastructure for a living. And it is not a failure of the approach. It is what the approach is supposed to do: automated evaluation is a filter that reduces 495 to 140, so that expensive human review runs on 140 rather than 495.
The final output was a curated ground truth dataset of 685 data points, split 60/40 for training and testing. With that, they achieved significant improvements in smaller model performance.
Six hundred and eighty-five examples. That is the other number worth sitting with, because it runs against the instinct that flywheels are about volume.
What a data flywheel actually is
The definition is straightforward: a feedback loop where data collected from interactions or processes is used to continuously refine AI models, which in turn generates better outcomes and more valuable data.
The compounding argument is what makes it attractive. More data improves the model, which improves the application, which produces higher business value, which attracts and retains more users, which generates more data.
The most cited analogy is Tesla: millions of miles driven, edge cases captured, fed back into training. The cars get better, the data gets better, the loop spins faster.
But the strategic claim underneath is more interesting than the loop diagram, and it is where the business case sits. The core thesis is that continuous improvement of AI agents does not require constantly upgrading to larger models. It requires systematic feedback loops that let smaller, cheaper, faster models match or approach the accuracy of larger ones.
That reframes the flywheel from a quality initiative into a cost programme. You are not chasing a better model. You are chasing the same accuracy at lower latency and lower total cost of ownership.
The loop, stage by stage
Capture. Inference logs, prompts, responses, retrieval traces, tool calls, latency and user feedback signals. Stored somewhere queryable, commonly Elasticsearch in the NVIDIA and MLRun reference architecture.
Monitor and detect. Performance, stability and resource usage tracked continuously, surfacing candidates for review rather than waiting for a complaint.
Automated triage. LLM-as-a-judge and online evaluation narrow the candidate pool. This is the 495 to 140 step.
Human review. Subject matter experts confirm which flagged cases are genuine and label them correctly. This is the 140 to 32 step, and it is where the ground truth actually gets made.
Curate the dataset. The confirmed cases become a training and evaluation set, split for both.
Customise. Fine-tuning through LoRA, p-tuning or supervised fine-tuning against the curated set.
Evaluate. Candidate models run against production logs and held-out data, using zero-shot, RAG and LLM-as-judge evaluation.
Redeploy. The improved model ships, and generates the next round of logs.
Orchestration is what turns this from a project into a system. In the reference implementations, an orchestrator wraps the whole loop, triggering evaluation and customisation workflows when logs meet defined conditions, and escalating to humans where the decision needs one.
Why speed is the actual constraint
There is an argument in the practitioner literature that I think is underrated, and it explains why manual flywheels fail rather than merely underperforming.
Manual processes break down at scale. By the time you have reviewed examples, labelled data, retrained and deployed, the production system has moved on. Requirements changed. New edge cases appeared. You are always catching up.
That is a different failure mode from "we did not have enough data." A flywheel that takes eleven weeks to complete a cycle is not a slow flywheel. It is a stationary one, producing improvements calibrated to a distribution that no longer exists.
The measure that matters is therefore cycle time, not dataset size. A loop that closes in days against 200 examples beats a loop that closes in a quarter against 20,000.
Where flywheels quietly go wrong
Four failure modes, none of which appear in the vendor diagrams.
Survivorship bias in the logs. Your production logs contain interactions from users who stayed. The user who asked something the system handled badly and never came back generates one bad log and then nothing. The user who found it useful generates hundreds. So the log distribution over-represents what already works, and the flywheel optimises hardest for the cases that needed the least help.
This is structural and it does not fix itself with volume. It needs deliberate counterweighting: sampling abandoned sessions, tracking first-interaction churn, and treating the absence of logs from a segment as a finding rather than an absence of data.
Automated triage treated as ground truth. The 140 to 32 funnel is the warning. If NVIDIA had fine-tuned on all 140 flagged cases, roughly three quarters of that training signal would have been wrong, and the model would have learned to correct routing decisions that were already correct. Automated evaluation narrows the search space. It does not make the judgement.
Training on your own outputs without verification. A flywheel that captures model responses, judges them with a model and trains on the result is a recursive loop with no external anchor. The model collapse literature is clear that recursive training on synthetic data degrades models, with rare cases disappearing first, and that the mitigation is verification and accumulation of real human-anchored data rather than replacement. Human review is not a quality nicety in this loop; it is the anchor that stops it eating itself.
Privacy and residency, which almost nobody addresses. Production logs are user data. They contain personal information, sometimes sensitive categories, and they are subject to the same cross-border transfer rules as any other personal data. A flywheel that captures logs in one jurisdiction and processes them for training in another is a data transfer, whether or not anyone in the pipeline calls it one. Consent language written for service delivery does not automatically cover model training, and retrofitting it is harder than getting it right at capture.
The limit nobody states clearly
Here is the structural constraint, and I think it is the most important thing to understand about flywheels before investing in one.
A flywheel amplifies what you already have. It cannot bootstrap what you do not.
If your product has no users in Indonesia, your logs contain no Indonesian interactions. The flywheel will not improve Indonesian performance, because there is nothing to feed it. And since the model performs poorly in Indonesian, adoption stays low, which means the logs stay empty. The loop runs in reverse: weak coverage produces weak usage produces no data produces continued weak coverage.
The same applies to any segment you serve badly enough that users leave, any domain you have not entered, and any edge case rare enough not to appear.
Which means a flywheel is an excellent mechanism for improving where you are already competent and a useless one for entering where you are not. Those require commissioned collection: data produced deliberately rather than harvested from traffic that does not exist.
This is where our own work sits, so I will declare the interest. Lifewood collects and annotates multilingual data across 50plus languages, and the pattern we see repeatedly is a client with a mature flywheel in two or three languages and flat performance in the markets they want to grow into. The flywheel is working exactly as designed. It just cannot manufacture logs from users who are not there yet.
The practical framing we use: flywheel for the languages you have, commissioned collection for the languages you want. Once a market reaches enough usage, the flywheel takes over and the collection cost stops. Before that, there is nothing to spin.
The human layer, and why 685 examples was enough
Return to the NVIDIA numbers, because they contain the most useful lesson in this whole area.
685 curated data points produced significant improvements in smaller model performance. Not 685,000. The value was concentrated entirely in the curation: the automated pass found candidates, and subject matter experts determined which were real and what the correct behaviour should have been.
That is the same finding that recurs throughout data work. A small, correct, well-targeted dataset outperforms a large, noisy one, and the cost sits in making it correct rather than making it large.
For a flywheel specifically, three human roles are load-bearing:
Subject matter experts who can judge whether a flagged failure is genuinely a failure. This requires domain knowledge, not annotation training.
Reviewers who define correct behaviour, not just identify wrong behaviour. Knowing the routing was wrong is half the label; knowing where it should have gone is the other half and the harder one.
Someone who owns the sampling strategy, because what gets reviewed determines what gets fixed, and automated flagging inherits whatever biases the judge model has.
None of these are jobs a general annotation pool does well, which is the same conclusion as reasoning trace work and preference data. The pattern is consistent: as the data gets closer to judgement, the annotator profile shifts from trained to qualified.
Getting started without building the full stack
The advice from practitioners running these systems is unusually concrete and worth following literally.
Pick one high-value improvement area. Safety, accuracy, or response style. Not all three.
Set up online evaluation for that specific concern, rather than general quality monitoring.
Collect 100 to 200 labelled examples. That is the recommended starting volume, and the NVIDIA case suggests the ceiling for useful results is lower than most teams assume.
Measure cycle time from log to redeploy as your primary operational metric.
Put a human confirmation step between automated flagging and dataset inclusion, always. The 140 to 32 ratio is the argument.
Instrument for what is missing, not just what failed. Abandoned sessions, unanswered queries, segments generating no traffic.
Sort out consent and residency at capture, because retrofitting a lawful basis for training use across a year of accumulated logs is considerably harder than writing it correctly on day one.
Key takeaways
- NVIDIA's internal agent flywheel took 495 unsatisfactory responses, flagged 140 as routing failures via LLM-as-ajudge, and subject matter experts confirmed only 32 as genuine, roughly a 77% false positive rate on automated triage.
- The resulting curated dataset was 685 data points, split 60/40 train and test, and produced significant improvements in smaller model performance.
- Automated evaluation is a filter that reduces the review population. It is not a judgement and should not be treated as ground truth.
- A data flywheel is a feedback loop where interaction data continuously refines models, which produce better outcomes and more valuable data.
- The strategic claim is that continuous improvement does not require larger models, but systematic feedback loops letting smaller models match larger ones at lower latency and cost.
- The loop runs: capture, monitor, automated triage, human review, curate, customise, evaluate, redeploy, with orchestration turning it from a project into a system.
- Cycle time is the binding constraint, not dataset size. Manual processes break down because by the time you have reviewed, labelled, retrained and deployed, the production distribution has moved on.
- Survivorship bias is structural: logs over-represent users who stayed, so the flywheel optimises hardest for cases that needed the least help.
- Training on model outputs judged by models with no human anchor is recursive, and the model collapse literature shows this degrades models with rare cases lost first.
- Production logs are user data subject to transfer and consent rules. Consent for service delivery does not automatically cover training use.
- A flywheel amplifies what you already have and cannot bootstrap what you do not. No users in a market means no logs, which means no improvement, which sustains low adoption.
- Flywheel for the languages and segments you have; commissioned collection for the ones you want to enter.
- Three human roles are load-bearing: subject matter experts who judge genuine failures, reviewers who define correct behaviour rather than just flagging wrong behaviour, and an owner of the sampling strategy.
- Practitioner starting advice: pick one improvement area, set up online evaluation for it specifically, and collect 100 to 200 labelled examples.
Sources and further reading
- ZenML LLMOps Database, "Nvidia: Data Flywheels for Cost-Effective AI Agent Optimization", on the NV Info Agent architecture, the 495 to 140 to 32 triage funnel, the 685-point curated dataset and the smaller-model thesis
- NVIDIA Glossary, "Data flywheel: What it is and how it works", on the definition, the AT&T deployment and the business objectives of flywheel programmes
- NVIDIA, "Build an Enterprise Data Flywheel" Blueprint, on the automated loop collecting production traffic logs, evaluating, fine-tuning and redeploying
- Iguazio, "Build Observable Data Flywheels for Production with MLRun and NVIDIA NeMo Microservices", on orchestration, log storage, NeMo Customizer techniques including LoRA and p-tuning, and NeMo Evaluator methods
- Iguazio Glossary, "What is a Data Flywheel?", on the compounding loop and the continuous improvement mechanism
- Arize AI, "Building the Data Flywheel for Smarter AI Systems with Arize AX and NVIDIA NeMo", on cycle time as the constraint, the manual process breakdown argument and the 100 to 200 example starting point
- "Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support", arXiv
- Shumailov et al., "AI models collapse when trained on recursively generated data", Nature, on recursive training degradation, discussed earlier in this series. DOI 10.1038/s41586-024-07566-y Lifewood, multilingual data collection and human-in-the-loop AI data services