Short answer. Four layers of it: screen-understanding data that teaches the model to read interfaces, grounding data that maps instructions to exact pixels, multi-step trajectories that teach how UI state evolves under actions, and reasoning traces plus reward signals that teach why each step happens. The supply mix spans expert human demonstrations, synthetic exploration, mined tutorials and instructional video — and the research keeps landing on the same conclusion: carefully verified human data outperforms much larger unverified corpora.
What is a computer-use agent actually learning?
A perception-to-action loop — see the screen, find the target, act, and track what changed — and the training data decomposes along exactly those joints.
A computer-use agent is a vision-language model asked to do something no web corpus teaches directly: look at a rendered interface, connect an instruction like "export the report as PDF" to one specific button among hundreds of elements, execute a low-level action — click, type, drag, scroll — and then understand the new screen that action produced, dozens of times in a row, without losing the plot. Each capability in that loop fails independently: an agent can describe a screen perfectly and still click 40 pixels left of the target; it can ground clicks precisely and still have no idea what sequence of clicks achieves a goal; it can execute a known sequence and collapse the moment a pop-up changes the state.
That is why GUI agent training data is not one dataset but a stack. The research community has converged on a decomposition that mirrors the loop: understanding data for perception, grounding data for targeting, trajectory data for multi-step execution, and rationale-and-reward data for planning and judgment.
Frameworks like ScaleCUA make the stack explicit, annotating a shared corpus of screenshots and metadata into exactly these task families before any training run begins — because a model's weakest layer, not its average, sets what it can do on a real desktop.
The four-layer GUI training stack 1 2 3 4 REASONING & REWARD UNDERSTANDING GROUNDING TRAJECTORIES Element descriptions, referring OCR, spatial layout, interface and screen-transition captioning Instruction-to-pixels: point grounding for clicks, bounding boxes for regions, action grounding for commands Multi-step episodes — screenshots, actions and parameters — showing how UI state evolves toward a goal Step-level rationales that explain the why, plus verifiable reward signals for RL post-training Each layer trains a different failure mode: perception, targeting, execution and judgment fail independently in deployed agents.
What are the four core data types?
Understanding teaches reading; grounding teaches aiming; trajectories teach doing; rationales and rewards teach deciding.
Screen-understanding data. Before an agent can act it has to parse: describe an element's appearance, extract its text (referring OCR), infer its function and the intent behind it, summarise a whole interface, and caption what changed between two screenshots. ScaleCUA's curation formalises these as element-level and screenshot-level tasks, including screen-transition captioning — the skill of noticing that a click opened a dialog — which underpins everything downstream.
Grounding data. Grounding maps language to screen coordinates, and it comes in three flavours: point grounding (the exact click location), bounding-box grounding (the region for selection-style operations), and action grounding (connecting a spatial target to the low-level command that operates it). The dataset ecosystem here is rich — SeeClick, UGround, OS-Atlas, Aria-UI and ScreenSpot-Pro, whose 1,581 highresolution tasks across 23 professional industries expose how much harder dense expert software is than consumer web pages. The standout for desktop is GroundCUA: built from expert human demonstrations across 87 applications in 12 categories, 56K screenshots with every on-screen element annotated — over 3.56 million human-verified annotations.
Trajectory data. Trajectories are complete episodes — instruction, screenshots, actions with parameters, step by step — and they are what teach an agent that interfaces are stateful. The canonical sets were built from human demonstrations: Mind2Web for web tasks, AITW for Android at scale, GUI-Odyssey's 7,700 mobile episodes spanning cross-app workflows, and AndroidControl and JEDI enriching steps with low-level action descriptions that bridge intent to executable operations. Robustness variants now add the messiness of reality: one benchmark collects 10,000+ action sequences under seven anomaly conditions — occlusion, dynamic content changes — because deployed screens rarely behave like clean captures.
Reasoning and reward data. The newest layer trains the why: step-level natural-language rationales that record what the demonstrator observed, planned and expected — the approach behind AITZ, AgentTrek, OSGenesis and Aguvis — and reward signals for reinforcement learning, from learned reward models to the partially verifiable rewards used in recent post-training work. TongUI shows the retrofit pattern: prompting a model to generate consistent "thoughts" for 256K existing samples, with visual markers on the clicked element so the rationale and the action cannot drift apart.
Where does the data come from — humans, synthesis, tutorials, video?
Four supply lines with opposite economics — and the field's clearest recent finding is that verified human data punches far above its size.
Human demonstrations set the fidelity bar. The founding datasets — Mind2Web, AndroidControl, GUIOdyssey — were built by people performing real tasks, which is why they capture natural interaction patterns rather than random clicking. GroundCUA's protocol is the modern template: annotators design everyday tasks reflecting common goals, carry them out in open-source applications, and every element gets human-verified labels. The cost is the constraint the whole field organises around — expert demonstration is expensive per episode — but the return is now measured: GroundNext, trained on that expert-driven data, matches or beats models trained on substantially more data, with the authors concluding that high-quality, expert-driven datasets play the critical role in advancing general-purpose computer-use agents.
Synthetic pipelines buy scale. To escape collection costs, OS-Genesis extracts trajectories via agentdriven exploration guided by learned reward models; WebSynthesis runs world-model-guided search over simulated web interfaces; GUI-ReWalk combines stochastic exploration with intent-aware reasoning; and internet-scale pipelines have produced 94K successful web trajectories across 49K URLs and 720K screenshots with no human annotation at all. The trade is explicit in the papers: synthetic data delivers coverage and volume, and needs verification layers precisely because nobody watched it being made.
Tutorials and video are the sleeper sources. The internet already contains millions of demonstrations of software use — they are just not labelled as training data. TongUI mines multimodal web tutorials into 143K executable trajectories; VideoAgentTrek converts 39,000 unlabelled instructional videos into 1.52 million reasoning-and-action steps — about 26 billion training tokens — by detecting actions in the footage and reconstructing the steps around them. Its final mix is a useful snapshot of current practice: roughly 26B tokens from video, 8B from harmonised human demonstrations, and 1B of focused grounding pairs — scale from found data, fidelity from human data, precision from grounding.
RL environments close the loop. Benchmarks like OSWorld double as training grounds: reinforcement learning post-training against executable environments — where task success is checkable — consistently improves agents beyond imitation, which makes verifiable task definitions and reward functions a data type of their own.
One real pretraining mix — and what the field learned about quality Video-derived interaction steps (VideoAgentTrek)
~26B tokens Harmonised human demonstrations (OpenCUA, Aguvis)
~8B tokens Focused GUI grounding pairs (OSWorld-G subset)
~1B tokens While quality beats bulk 3.56M human-verified element annotations in GroundCUA, from expert demonstrations across 87 desktop apps ≥ parity GroundNext, trained on that expert data, matches or beats models trained on substantially more data 7,700 human-demonstrated mobile episodes in GUIOdyssey, including cross-app workflows synthesis still struggles to fake Token mix from VideoAgentTrek's reported corpus; quality findings from the GroundCUA/GroundNext papers.
What does quality mean here, and how is it produced?
Pixel-accurate, state-aware, verified by a second human — and covering the platforms, industries and languages real desktops actually run.
The error tolerances are brutal. In most annotation work a slightly loose label degrades a statistic; in GUI data, a bounding box drawn 20 pixels wide teaches an agent to click the wrong control, and one mislabelled step in a trajectory poisons everything after it, because each action conditions the next state. That is why the strongest datasets describe their pipelines in terms of expert demonstrators, per-element verification and screenshot-level review — and why "human-verified" appears in dataset abstracts as a selling point, not a footnote.
Coverage is a quality dimension, not a bonus. Agents trained on consumer web pages stumble in dense professional software — the gap ScreenSpot-Pro's 23-industry benchmark was built to expose — and corpora now deliberately span Windows, macOS, Android and web, plus anomaly conditions like occlusion and dynamic content. The quietest gap is linguistic: interfaces exist in every script and locale, tutorials and demonstrations skew heavily English, and an agent that has only ever grounded English labels has never really seen most of the world's screens.
This is demonstration work at industrial scale — which is where the supply chain comes in.
Behind every "expert-driven" dataset is the operational problem of recruiting people who know the software, having them perform natural tasks, annotating every element they touched, and independently verifying the result. That is, precisely, AI data-service work — and it is the shape of what Lifewood supplies from its side of the industry: task demonstration and data collection run through delivery centres in 30+ countries, elementlevel annotation across text, image and interaction data, coverage in 50+ languages for the locales most corpora miss, and the company's dual-layer human-in-the-loop review — one pass produces the demonstration, an independent pass verifies every label against the screen — applied at the per-element standard this category demands. One more production rule matters here: screenshots of real work capture real information, so PII scrubbing and consent belong in the pipeline before a single frame reaches a training run.
A caution on the numbers. The dataset sizes, token counts and findings above come from the cited papers and repositories and describe those specific corpora and models; results like "matches models trained on more data" are benchmark-specific claims from the authors, and the field moves fast enough that this year's state of the art is next year's baseline. Check the papers — all are openly available — before building on any specific figure.
Key takeaways
- Computer-use agents learn a loop — read the screen, find the target, act, track the change — and training data decomposes along the same joints: understanding, grounding, trajectories, and reasoningplus-reward.
- Understanding data covers element descriptions, referring OCR, layout and screen-transition captioning; grounding data maps instructions to points, boxes and actions — with ScreenSpot-Pro showing how much harder dense professional software is than the consumer web.
- Trajectory data teaches statefulness: human-demonstrated corpora like Mind2Web, AITW, GUI-Odyssey (7,700 cross-app episodes) and AndroidControl remain the fidelity standard, now extended with anomaly conditions like occlusion and dynamic content.
- Reasoning traces (AITZ, OS-Genesis, Aguvis, TongUI's generated thoughts) and verifiable RL rewards form the newest layer — teaching why a step happens and letting post-training improve on imitation.
- • Supply comes from four lines with opposite economics: costly high-fidelity human demonstrations, scalable synthetic exploration (up to 94K trajectories with no human labels), mined tutorials (TongUI's 143K), and instructional video (VideoAgentTrek's 39K videos → ~26B tokens).
- The field's clearest recent finding favours quality: GroundCUA's 3.56M human-verified annotations trained models that match or beat systems trained on substantially more data.
- Quality in this category means pixel-accurate labels, state-consistent trajectories, independent human verification, cross-platform and cross-industry coverage — and closing the multilingual gap most corpora ignore.
- Producing it is industrial demonstration-and-annotation work: recruit users who know the software, capture natural tasks, annotate per element, verify with a second pass, and scrub PII before anything trains.
- All figures are paper-specific and the field moves quarterly — read the cited papers before building on any number.
Sources and further reading
- - Liu et al., "ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data", on the understanding/ grounding/trajectory task decomposition and cross-platform corpus curation
- - Feizi et al., "Grounding Computer Use Agents on Human Demonstrations" (GroundCUA/GroundNext), on the 87-app, 3.56M human-verified annotation dataset and the quality-over-quantity result
- - "GUI-Libra: Training Native GUI Agents to Reason and Act", for its survey of grounding, trajectory and rationale datasets and partially verifiable RL
- - "GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration", on human-demonstrated founding datasets, collection costs and synthetic-generation approaches
- - Xu et al., "VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos", on converting 39K videos into ~26B training tokens and its human-demo and grounding data mix
- - Zhang et al., "TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials", on tutorial mining and generating action-consistent reasoning traces for 256K samples
- - "OmniActor: A Generalist GUI and Embodied Agent", for a concrete grounding-then-trajectory training recipe and its ~3.4M/~4.1M sample scales
- - "CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents", on human-annotated video demonstration corpora
- - Computer/Browser/Phone-Use Agent Datasets (curated repository), for dataset statistics including ScreenSpot-Pro, ShowUI-desktop, anomaly-condition sets and internet-scale synthetic trajectories
- Computer-Browser-Phone-Use-Agent-Datasets.
- - Lifewood, AI data collection, annotation and dual-layer human-in-the-loop verification services