Part 6 of our series on the data problem in Vision-Language-Action research. [Part 1: the bottleneck overview · Part 2: the language gap · Part 3: the failure data · Part 4: the evaluation crisis · Part 5: the synthetic trust layer]
The single most exciting number in the VLA corpus is also the most misleading. HumanNet assembles roughly one million hours of egocentric human video (arXiv:2605.06747). Being-H0.5 works from 35,000 hours more (arXiv:2601.12993). Set against the thousands of hours of teleoperated robot data that represent the field's hard-won total, these corpora look like the end of the data bottleneck. Humans manipulate objects all day, on camera, for free. Why teleoperate a robot when YouTube has already recorded the demonstrations?
Because that video is action-free. It shows what happened; it doesn't record what to do. Bridging that gap, turning pixels of a hand into a signal your policy can learn from, is one of the largest annotation problems in robotics. The million hours are not a dataset. They are raw material, and this post is about the labor that turns raw material into training data.
A teleoperated robot episode contains a privileged thing that raw human video lacks: a synchronized stream of actions. Every frame is paired with the exact motor commands that produced it, in the robot's own action space. That pairing is what behavioral cloning learns from.
Human video has none of it. You see a hand approach a mug and lift it, but there are no joint angles, no gripper commands, no action labels, and the hand isn't even your robot's end effector. The correspondence problem, mapping a five-fingered human hand onto a parallel-jaw gripper and human kinematics onto robot kinematics, sits between the video and any usable action. The video is a demonstration of what to do with no record of how, in a body that isn't the robot's.
This is why an entire sub-literature has grown up specifically to extract action from passive video. Latent-action models and inverse-dynamics methods try to infer, from consecutive frames, what action must have occurred between them. That work is, in effect, an automated labeling effort: its whole purpose is to synthesize the action annotations human video doesn't carry. The existence and size of that sub-field is the clearest evidence that action-labeling human video is a first-class problem, not a preprocessing detail.
When groups actually try to use human video, they hit the same wall, and they name it. The blocker is not the quantity of video; it's the absence of task-aligned annotations on it (arXiv:2606.00054). Hours of footage with no segmentation into tasks, no grounding of instructions to spans, no labeled primitives, cannot be pointed at a training objective. The signal is in there, but nothing has aligned it to the task.
EgoVLA (arXiv:2507.12440) is instructive here. It treats egocentric human video as a genuine training source for manipulation policies; the promise is real. But doing so requires exactly the grounding work that raw video lacks: relating human hand motion to robot actions and aligning it to tasks. The bet that human video can substitute for robot data is a bet on the annotation pipeline that stands between them. The video is necessary; it is nowhere near sufficient.
This connects straight back to Part 2. There, the problem was that robot datasets have impoverished language; here, human video has no task language until someone adds it. Both are the same underlying gap, the missing semantic layer that tells a model what a segment of behavior means, and both close the same way: by annotation.
If you're evaluating human video as a data source, it helps to see the full stack of labels required to make it trainable. Each layer is a distinct annotation task, and each adds value the layer below can't.
Episode and task segmentation. Continuous video has to be cut into meaningful units (this span is "open the drawer," this span is "retrieve the cup") before anything downstream can use it. Long unsegmented footage is the least trainable form the data can take.
Hand and object pose. Recovering hand pose, contact events, and manipulated-object trajectories converts pixels into geometry, the substrate the correspondence problem needs to map human motion onto a robot.
Action and primitive labels. The task-aligned primitives (reach, grasp, transport, place, release) that give a policy a vocabulary of behavior, aligned to the instruction rather than left implicit in the frames.
Intention-aligned language. Instructions grounded to the specific span they describe, at the fine granularity Part 2 showed models are starving for: not "person makes coffee" over ten minutes, but discriminative, execution-level language tied to each primitive.
OpenEgo is a useful worked example of what a serious schema for this looks like, defining structured annotations over egocentric manipulation video rather than leaving it as undifferentiated footage. The lesson isn't the specific schema; it's that the schema is the product. The raw hours are commodities. The annotation layer is the asset.
The obvious objection: can't latent-action and inverse-dynamics models label this automatically? Partly, and they should be used. But the same caution from Parts 2 and 5 applies here.
Automated action inference produces silent failures, plausible-looking labels that are subtly wrong, and the fine-grained spatial judgments that matter most (contact points, approach directions, hand-object relationships) are precisely where generic 2D-pretrained vision-language models are weakest (arXiv:2604.18484). And the human-to-robot correspondence, deciding what a parallel-jaw gripper should do given what a human hand did, is a judgment call about intent, not a measurement you can read off the frames. The pattern that keeps recurring across this series holds again: automate the volume, verify the judgment. The pipelines that work put human verification exactly where the automated labels are least reliable.
For a team eyeing egocentric video as a way past the robot-data bottleneck, the corpus suggests a clear-eyed plan.
Treat the video as raw material with a known conversion cost, not as a finished dataset. The million hours are the cheap part; budget for the annotation layer that makes them trainable, because that's the part that determines whether any of it works.
Decide how far up the annotation stack you need to go. Segmentation alone unlocks some uses; hand-pose and intention-aligned primitives unlock far more. The value climbs with the annotation depth, and so does the effort.
Use automated action inference for scale and human verification for trust, concentrating human effort on correspondence and fine-grained spatial labels where auto-labelers demonstrably fail.
And borrow the schema thinking from OpenEgo rather than inventing structure ad hoc. A consistent annotation schema applied across the whole corpus is worth more than a richer schema applied unevenly, the same lesson the failure taxonomies taught in Part 3.
A million hours of human video is one of the most valuable raw resources in robotics. It is also inert. The teams that win with it will be the ones who treat labeling not as the tax they pay to use the data, but as the step that creates the data in the first place.
Next in the series: tactile, force, and audio, the modalities the corpus has barely touched.
This post draws on our analysis of 1,228 VLA papers from arXiv cs.RO (Feb 2023 – Jun 2026). See Part 1 for methodology.