Short answer. Four layers of it: screen-understanding data that teaches the model to read interfaces, grounding data that maps instructions to exact pixels, multi-step trajectories that teach how UI state evolves under actions, and reasoning traces plus reward signals that teach why each step happens. The supply mix spans expert human demonstrations, synthetic exploration, mined tutorials and instructional video — and the research keeps landing on the same conclusion: carefully verified human data outperforms much larger unverified corpora.
Key takeaways
- Computer-use agents learn a loop — read the screen, find the target, act, track the change — and training data decomposes along the same joints: understanding, grounding, trajectories, and reasoning-plus-reward.
- Trajectory data teaches statefulness: human-demonstrated corpora such as Mind2Web, AITW and GUI-Odyssey's 7,700 cross-app episodes remain the fidelity standard, now extended with anomaly conditions like occlusion and dynamic content.
- Supply comes from four lines with opposite economics: costly high-fidelity human demonstrations, scalable synthetic exploration, mined tutorials, and instructional video.
- The field's clearest recent finding favours quality: GroundCUA's 3.56 million human-verified annotations trained models that match or beat systems trained on substantially more data.
- Quality in this category means pixel-accurate labels, state-consistent trajectories, independent human verification, cross-platform coverage, and closing the multilingual gap most corpora ignore.
What is a computer-use agent actually learning?
A computer-use agent learns a perception-to-action loop: see the screen, find the target, act, and track what changed. Training data for it decomposes along exactly those joints rather than existing as one dataset.
A computer-use agent is a vision-language model asked to do something no web corpus teaches directly: look at a rendered interface, connect an instruction like "export the report as PDF" to one specific button among hundreds of elements, execute a low-level action — click, type, drag, scroll — and then understand the new screen that action produced, dozens of times in a row, without losing the plot. Each capability in that loop fails independently: an agent can describe a screen perfectly and still click 40 pixels left of the target; it can ground clicks precisely and still have no idea what sequence of clicks achieves a goal; it can execute a known sequence and collapse the moment a pop-up changes the state.
That is why GUI agent training data is a stack, not a single dataset: understanding data for perception, grounding data for targeting, trajectory data for multi-step execution, and rationale-and-reward data for planning and judgment. Frameworks like ScaleCUA make the stack explicit, annotating a shared corpus of screenshots and metadata into exactly these task families before any training run begins — because a model's weakest layer, not its average, sets what it can do on a real desktop. Each layer trains a different failure mode: perception, targeting, execution and judgment fail independently in deployed agents.
What are the four core data types?
Understanding data teaches reading, grounding data teaches aiming, trajectory data teaches doing, and rationale-and-reward data teaches deciding.
Screen-understanding data covers describing an element's appearance, extracting its text (referring OCR), inferring its function and intent, summarising a whole interface, and captioning what changed between two screenshots. ScaleCUA's curation formalises these as element-level and screenshot-level tasks, including screen-transition captioning — noticing that a click opened a dialog — which underpins everything downstream.
Grounding data maps language to screen coordinates in three flavours: point grounding (the exact click location), bounding-box grounding (the region for selection-style operations), and action grounding (connecting a spatial target to the low-level command that operates it). The dataset ecosystem here is rich — SeeClick, UGround, OS-Atlas, Aria-UI and ScreenSpot-Pro, whose 1,581 high-resolution tasks across 23 professional industries expose how much harder dense expert software is than consumer web pages. The standout for desktop is GroundCUA: built from expert human demonstrations across 87 applications in 12 categories, 56K screenshots with every on-screen element annotated, totalling over 3.56 million human-verified annotations.
Trajectory data consists of complete episodes — instruction, screenshots, actions with parameters, step by step — and teaches an agent that interfaces are stateful. The canonical sets were built from human demonstrations: Mind2Web for web tasks, AITW for Android at scale, GUI-Odyssey's 7,700 mobile episodes spanning cross-app workflows, and AndroidControl and JEDI enriching steps with low-level action descriptions that bridge intent to executable operations. Robustness variants now add the messiness of reality: one benchmark collects over 10,000 action sequences under seven anomaly conditions, including occlusion and dynamic content changes, because deployed screens rarely behave like clean captures.
Reasoning-and-reward data is the newest layer, training the why behind each step: step-level natural-language rationales that record what the demonstrator observed, planned and expected — the approach behind AITZ, AgentTrek, OS-Genesis and Aguvis — and reward signals for reinforcement learning, from learned reward models to the partially verifiable rewards used in recent post-training work. TongUI shows the retrofit pattern: prompting a model to generate consistent "thoughts" for 256K existing samples, with visual markers on the clicked element so the rationale and the action cannot drift apart.
Where does the training data come from?
Four supply lines carry opposite economics, and the field's clearest recent finding is that verified human data punches far above its size compared with unverified volume.
Human demonstrations set the fidelity bar. The founding datasets — Mind2Web, AndroidControl, GUI-Odyssey — were built by people performing real tasks, which is why they capture natural interaction patterns rather than random clicking. GroundCUA's protocol is the modern template: annotators design everyday tasks reflecting common goals, carry them out in open-source applications, and every element gets a human-verified label. The cost is the constraint the whole field organises around — expert demonstration is expensive per episode — but the return is now measured: GroundNext, trained on that expert-driven data, matches or beats models trained on substantially more data, with the authors concluding that high-quality, expert-driven datasets play the critical role in advancing general-purpose computer-use agents.
Synthetic pipelines buy scale. OS-Genesis extracts trajectories via agent-driven exploration guided by learned reward models; WebSynthesis runs world-model-guided search over simulated web interfaces; GUI-ReWalk combines stochastic exploration with intent-aware reasoning; and internet-scale pipelines have produced 94K successful web trajectories across 49K URLs and 720K screenshots with no human annotation at all. The trade is explicit in the papers: synthetic data delivers coverage and volume, and needs verification layers precisely because nobody watched it being made.
Tutorials and video are the sleeper sources. The internet already contains millions of demonstrations of software use that are simply not labelled as training data. TongUI mines multimodal web tutorials into 143K executable trajectories; VideoAgentTrek converts 39,000 unlabelled instructional videos into 1.52 million reasoning-and-action steps — about 26 billion training tokens — by detecting actions in the footage and reconstructing the steps around them. Its final mix is a useful snapshot of current practice: roughly 26 billion tokens from video, 8 billion from harmonised human demonstrations, and 1 billion of focused grounding pairs — scale from found data, fidelity from human data, precision from grounding.
RL environments close the loop. Benchmarks like OSWorld double as training grounds: reinforcement learning post-training against executable environments, where task success is checkable, consistently improves agents beyond imitation, which makes verifiable task definitions and reward functions a data type of their own. Reading how reasoning trace data teaches models to show their work helps explain why the rationale layer above matters as much as the action labels.
What does quality mean for this kind of data, and how is it produced?
Quality means pixel-accurate labels, state-consistent trajectories, independent human verification, and coverage across the platforms, industries and languages real desktops actually run.
The error tolerances are brutal. In most annotation work a slightly loose label degrades a statistic; in GUI data, a bounding box drawn 20 pixels wide teaches an agent to click the wrong control, and one mislabelled step in a trajectory poisons everything after it, because each action conditions the next state. That is why the strongest datasets describe their pipelines in terms of expert demonstrators, per-element verification and screenshot-level review — and why "human-verified" appears in dataset abstracts as a selling point, not a footnote. Consistent labelling standards are also the subject that inter-annotator agreement metrics exist to measure, and the same statistical logic applies to GUI element labels as to any other annotation task.
Coverage is a quality dimension, not a bonus. Agents trained on consumer web pages stumble in dense professional software — the gap ScreenSpot-Pro's 23-industry benchmark was built to expose — and corpora now deliberately span Windows, macOS, Android and web, plus anomaly conditions like occlusion and dynamic content. The quietest gap is linguistic: interfaces exist in every script and locale, tutorials and demonstrations skew heavily English, and an agent that has only ever grounded English labels has never really seen most of the world's screens.
Behind every "expert-driven" dataset is the operational problem of recruiting people who know the software, having them perform natural tasks, annotating every element they touched, and independently verifying the result. That is, precisely, AI data-service work, and it is the shape of what Lifewood supplies from its side of the industry: task demonstration and data collection run through delivery centres across 40+ delivery centres in 30+ countries, element-level annotation across text, image and interaction data, coverage in 50+ languages for the locales most corpora miss, and a dual-layer human-in-the-loop review — one pass produces the demonstration, an independent pass verifies every label against the screen — applied at the per-element standard this category demands. Programs at this scale also depend on how the broader agentic AI training data stack is assembled and QA'd before it ever reaches a model, and on the multimodal annotation pipelines that combine screenshots, text and interaction logs into one labelled record. One more production rule matters here: screenshots of real work capture real information, so PII scrubbing and consent belong in the pipeline before a single frame reaches a training run.
A caution on the numbers: the dataset sizes, token counts and findings above come from the cited papers and repositories and describe those specific corpora and models. Results such as "matches models trained on more data" are benchmark-specific claims from the authors, and the field moves fast enough that this year's state of the art is next year's baseline. The cited papers are open and worth checking before building on any specific figure.