Skip to main content
← All articles
AI Data

Annotation for Robotics and Physical AI: Manipulation, Egocentric Video and Affordance

Short answer. Robotics annotation is not autonomous-driving annotation at a larger scale — it is a different job.

Mumu D. · August 2026 · 10 min read

Download PDF

Short answer. Robotics annotation is not autonomous-driving annotation at a larger scale — it is a different job.

Driving data asks you to name what is in a scene. Robot data asks you to describe what a pair of hands is doing to an object, frame by frame, in three dimensions, from the actor's own viewpoint. That means dexterous hand-pose labels, contact-state transitions, task and sub-task segmentation, and affordance annotation that says which part of an object can be acted on and how. It is slower, more expensive, and considerably harder to get right. It is also where demand is growing fastest.

Why did robotics data suddenly get so big?

Because humanoids stopped being demos. Once robots go into real warehouses on real production schedules, the bottleneck moves from model architecture to data — and the data has to be collected by people.

The scale-up is visible from several directions at once. Figure AI reported in January 2026 that its BotQ facility had delivered more than 350 Figure 03 units and lifted production from one robot a day to one an hour. Tesla began Optimus Gen 3 production at Fremont the same month. Analysts covering the sector are blunt about what this changed: in 2026 the constraint on enterprise physical-AI programmes is no longer architecture or compute, it is data quality and distribution coverage — and models trained on carefully curated real-world demonstration data have outperformed models trained on simulation alone, even at ten times the simulated scale.

Interest is climbing on the demand side too. US monthly search volume for "physical AI" went from roughly 1,900 in May 2025 to 6,600 by April 2026 — a 3.5× jump in twelve months, driven by humanoid programmes, open-source policy releases and the arrival of factory-style data pipelines in place of passive web scraping. What buyers are actually procuring, according to the marketplaces serving them, is egocentric video, teleoperation traces, manipulation demonstrations and evaluation sets, with commercial rights and consent artefacts attached.

At Lifewood we have watched this arrive as a shift in the questions clients ask. Two years ago a robotics enquiry meant LiDAR and bounding boxes. Now it starts with wearables, hand tracking and contact states — and almost always ends with a question about how many countries and kitchens we can collect in.

What does egocentric annotation actually involve?

Labelling what the robot will see. That single constraint reshapes everything downstream.

The reasoning is straightforward once you hear it. Models trained on third-person footage learn to recognise actions from the outside; models trained on first-person footage learn to perform them. A third-person camera shows the scene but misses the hands, misses the moment of contact, and misses the exact pixels a robot will see as it reaches. Egocentric video is naturally aligned with the perspective of a robot-mounted camera, which is why it has become the efficient way to expand manipulation datasets without buying more robots.

The datasets that resulted are enormous, and one in particular reset expectations.

Apple built EgoDex with the Vision Pro: 829 hours of 30 Hz egocentric video across 194 tabletop manipulation tasks, with SE(3) annotations for 25 joints of both hands in every frame, tracked on-device using calibrated cameras and visual-inertial SLAM. It carries language annotation, camera extrinsics and dexterous annotation together — a combination none of the earlier datasets offered.

Others took the opposite route and went to work. Egocentric-1M, released in April 2026, captured 2,153 factory workers across real industrial sites: 1.08 billion frames at 1080p and 30fps, 16.4TB, the first egocentric dataset collected exclusively in factories rather than homes or labs. EgoVerse, from a consortium including Georgia Tech, Stanford, UC San Diego, ETH Zürich, MIT and Meta Reality Labs, contributed 1,362 hours across 1,965 tasks, 240 scenes and 2,087 demonstrators from multiple countries, with multi-robot co-training showing gains of up to 30% relative improvement across embodiments. And EgoScale demonstrated a log-linear scaling law between human data volume and validation loss, with that loss correlating strongly to downstream robot performance.

That last finding is the commercial one. It says, in effect, that more annotated human demonstration reliably buys you a better robot — which is why capacity has become the constraint.

What robotics annotation asks for that driving annotation never did WHAT THE ANNOTATOR PRODUCES WHY IT MAT TERS TO THE POLICY DIFFICULTY HAND POSE Per-frame 3D joint positions for both hands, finger by finger Fine-grained precision that wrist-only or gripper tracking cannot supply Hardware-assisted, review-heavy CONTACT STATE The exact frame where contact begins and ends, per object A policy that cannot detect contact from its own viewpoint fails at placement Genuinely ambiguous GAZE AND PROXIMITY Where the demonstrator looked; gripper-to-target spatial relations Approach-phase errors cause grasp failures Needs synced capture TASK SEGMENTATION Atomic action boundaries and natural-language descriptions Lets language-conditioned policies map instructions to motion Guideline-sensitive AFFORDANCE Which region of an object affords which action, and with which grasp Enables generalisation to objects never seen in training The hardest of the set LABEL TYPE These labels are produced from the same footage, but each has its own failure mode, its own guideline and its own reviewer. Treating them as one task is the most common mistake we see.

Why is affordance labelling the hard part?

Because the definition itself is contested, and most datasets get it wrong in the same three ways.

Affordance is the concept of action possibility — what a given object permits a given actor to do, based on the object's physical properties and the actor's motor capacity. Useful in principle. Slippery in practice. Researchers at the University of Tokyo, proposing an annotation scheme for egocentric action video, identified three recurring problems in existing datasets: they mix up affordance with object functionality; they confuse affordance with goal-related action; and they ignore human motor capacity altogether. Their proposed fix combines goal-irrelevant motor actions with grasp types as the label, and adds the notion of mechanical action to capture what is possible between two objects.

Read that as an annotation brief and the implications land quickly. A knife's functionality is cutting. Its affordances include being gripped by the handle, pinched at the blade for a handover, and pressed down with the palm. Those are different labels, and a guideline that does not separate them will produce a dataset where "knife" means whatever each annotator assumed it meant that day. The broader field acknowledges the shortfall directly: affordance research still faces data scarcity, poor generalisation and difficulty deploying to the real world, with a specific lack of large-scale affordance datasets carrying precise segmentation maps.

Automated affordance extraction from egocentric video is advancing, and it should be used. But it inherits the same definitional problem — an automatic pipeline is only as coherent as the label schema it was built against. This is where the human layer earns its cost.

WHAT WORKS IN PRACTICE WHERE PROGRAMMES COME UNSTUCK

  • Separate guidelines per label type, not one document for all five

  • Treating affordance as a synonym for object function

  • Grasp taxonomy agreed and illustrated before collection starts

  • Ambiguous contact frames resolved silently

  • Contact-state adjudication by a second reviewer

  • Demonstrator pools drawn from one country or one body type

  • Synced multimodal capture: video, depth, audio, gaze

  • Diverse settings and demonstrators — scenes, kitchens, factories, countries

  • Consent artefacts and commercial rights captured at source

  • One annotator labelling all five types on the same clip

  • Lab-only collection that never sees a real workspace

  • PII in first-person footage discovered after delivery Robots trained on narrow demonstrator diversity generalise about as well as you would expect.

Human demonstrators are the sensor. Recruit and calibrate them like one.

What should teams get right before scaling?

The unglamorous parts, mostly — and earlier than feels necessary.

Two constraints deserve naming because they surprise people. The first is privacy: egocentric footage records whatever the wearer looked at, including faces, screens and documents nobody consented to. End-to-end PII removal, compliant storage and full audit trails have become table stakes for enterprise buyers, and ISO 27001 and SOC 2 are now baseline requirements rather than differentiators in robotics procurement. Retrofitting that after collection is painful and sometimes impossible.

The second is diversity of setting and demonstrator. A dataset collected in one lab by twenty graduate students produces a policy that works in that lab. The datasets driving real progress went the other way — thousands of demonstrators, hundreds of scenes, multiple countries. That is a logistics problem before it is an annotation problem, and it is precisely the kind of work our delivery network across 30+ countries was built for: recruiting demonstrators in genuinely different kitchens, workshops and warehouses, with native-language briefing so that task instructions mean the same thing everywhere.

Write the affordance schema before you collect anything. Separate function, goal-related action and motor affordance explicitly, and fix your grasp taxonomy up front.

Split the five label types across specialised reviewers. Hand pose, contact, gaze, segmentation and affordance each need their own guideline and their own gold set.

Adjudicate contact frames. The precise frame where contact begins is the most contested label in the whole pipeline; give it a second pair of eyes and a written tie-break rule.

Design demonstrator diversity deliberately. Handedness, hand size, height, working style, culture and setting all propagate into the policy.

Capture consent and rights at the moment of capture. Commercial-use rights and contributor consent are far cheaper to record than to reconstruct.

Run PII removal as part of the pipeline, not after it. First-person footage sees more than the task.

Use automated pre-labels for pose, humans for contact and affordance. Play to what each is reliably good at.

Keep sim and real honest against each other. Curated real demonstration data has outperformed simulation-only training even at ten times the scale.

Key takeaways

  • Robotics annotation is a different discipline from driving annotation, not a bigger version of it.
  • The bottleneck in 2026 physical-AI programmes is data quality and coverage, not architecture or compute.
  • Egocentric video matters because it matches what a robot-mounted camera sees: first-person footage teaches performing, third-person teaches recognising.
  • EgoDex set a new bar — 338K trajectories, 194 tasks, 90M frames, with per-frame SE(3) annotation for 25 joints of both hands.
  • EgoScale showed a log-linear scaling law between human data volume and validation loss, with loss correlating to robot performance.
  • Five label types come off the same footage — hand pose, contact state, gaze and proximity, task segmentation, affordance — and each needs its own guideline.
  • Affordance is the hardest: datasets routinely confuse it with object function or goal-related action, and ignore human motor capacity.
  • Curated real demonstration data has beaten simulation-only training even at ten times the simulated scale.
  • PII handling, consent artefacts, ISO 27001 and SOC 2 are baseline requirements in robotics procurement, not differentiators.

Sources and further reading

Frequently asked questions

For pre-training, often yes — EgoDex, Ego4D, EgoVerse and Egocentric-1M are substantial. For a specific robot in a specific workspace, no. Open datasets rarely contain your objects, your tooling or your failure cases, and cross-embodiment transfer still benefits from targeted collection.

Hand-pose tracking is largely hardware-assisted and automated pipelines for affordance extraction from egocentric video are improving. Contact-state boundaries and affordance schemas still need human judgement, because the difficulty there is definitional rather than perceptual.

It is growing quickly — forecast at a 41.3% CAGR — but as a complement. Real demonstration data has outperformed simulation-only training even when simulation was scaled ten times higher.

Multimodal synchronisation, 3D and temporal precision, specialist reviewers per label type, and the collection logistics of putting wearables on real people in real settings. It is closer to running a film shoot than a labelling queue.

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team