How do you build a diverse egocentric dataset?
HumanSignal Team
Egocentric capture has become cheap enough that hours are no longer the constraint. A head-mounted rig and a willing participant will produce more first-person video than a team can annotate, and the resulting dataset can still fail to teach a model anything about the conditions it will meet in deployment. The reason is that scale and diversity are bought separately, and capture that scales easily tends to scale along one axis while staying flat on the rest. This piece names the axes that vary independently, shows what the published datasets cover, and sets out how to specify diversity before a camera is switched on.
Two of the largest egocentric datasets illustrate the split cleanly, and neither is deficient; they were built for different purposes.
EgoDex, collected with Apple Vision Pro, contains 829 hours of 1080p video at 30 Hz, 338,000 episodes across 194 tasks, roughly 90 million frames, with paired 3D hand and finger tracking captured at recording time. By the authors' own description it consists entirely of active tabletop manipulation, ranging from tying shoelaces to folding laundry. That is enormous depth within one capture condition, and for learning dexterous hand motion it is exactly the right shape.
Ego4D made the opposite trade. It gathered 3,670 hours from 923 unique participants across 74 locations in 9 countries, with participants ranging in age from 18 to over 80 and spanning scenarios including household, outdoor, workplace, and leisure activity. Fewer hours per condition, far more conditions.
A model trained only on the first will encounter deployment conditions it has never seen. A model trained only on the second may never see any single task often enough to learn it precisely. Neither dataset is wrong, and a team that treats "hours collected" as the headline number will not notice which of the two it has built.
Diversity is not one quantity. Treating it as a single dial is what produces datasets that are broad in a way nobody needed.
How many distinct people, and how varied in body size, handedness, height, age, and skill level. Hand geometry and reach envelope differ enough between participants to change what a manipulation policy learns. Ego4D's 923 participants across an 18 to 80-plus age range is the high-water mark here; many task-focused sets use fewer than ten.
Distinct physical spaces, not distinct recordings in one space. A dataset shot in four rooms of one building has one lighting design, one furniture vocabulary, and one set of surface materials. Geography adds another layer, because household object inventories and room layouts differ substantially between regions.
Whether the dataset covers one activity class deeply or many shallowly, and whether tasks include multi-step sequences with dependencies rather than isolated actions. This axis is where EgoDex and Ego4D separate most sharply.
Headset versus chest mount versus glasses, field of view, frame rate, resolution, and whether depth or hand tracking is captured natively. A model trained on one rig's intrinsics and mounting height learns that viewpoint along with the task, which becomes visible only when the hardware changes.
Time of day, artificial versus natural light, occlusion, background motion, and whether the participant ever fails or corrects. Deliberately capturing failure is the least common practice and among the most useful, since deployment is where things go wrong.
These five are worth tracking separately because they trade against each other rather than moving together. Recruiting more participants usually means fewer hours each. Adding buildings raises travel and setup cost per hour. Widening the task set reduces episodes per task. A protocol that asks for all five at once without naming which dominates will quietly optimize for whichever is cheapest to satisfy, which is almost always more hours in the sites already booked.
Set the numbers side by side and the pattern is consistent: datasets are strong on the axes their builders needed and thin on the rest. This matters when you are evaluating whether to reuse a public dataset rather than commission your own, because the headline hours figure tells you almost nothing about whether the coverage you need is present. EgoDex is exceptional on task count and hand-tracking fidelity within one environment class. Ego4D is exceptional on participants and geography with less per-task density.
It is worth a comparison from outside egocentric capture entirely. DROID, which is teleoperated robot data rather than first-person human video, was built with scene diversity as an explicit design goal: 564 scenes across 52 buildings, covering 86 tasks, collected by 50 collectors over twelve months. The lesson transfers even though the modality does not. Scene count rose because it was a stated target of the protocol, not because collection happened to spread out.
This is the concrete form of the argument that robot foundation models are short on condition diversity as a named missing property rather than on volume, and it is the same reason scraped video does not substitute for designed capture. Found footage carries whatever distribution it happened to have.
State a number per axis before collection: distinct participants, distinct buildings, task classes, rig configurations, and lighting conditions. A protocol that says "varied household settings" produces whatever the recruiters found convenient. A protocol that says "no more than 15% of hours from any single building" produces a measurable property.
Track coverage per axis during collection and route new sessions toward the thin cells. Opportunistic capture concentrates on whoever is available and whichever spaces are easiest to book, which correlates with easy lighting, tidy rooms, and cooperative objects. Quota sampling costs more per hour and is the only way to prevent the correlation.
Every episode should carry participant identifier, environment identifier, task class, rig configuration, and condition flags. Without these fields you cannot measure coverage, rebalance a dataset, or hold out a condition for evaluation.
The held-out condition is the part teams skip and later want. Reserving one building, one participant group, or one lighting regime entirely from training is what lets you measure whether a policy generalizes across that axis rather than within it. That split has to be decided before collection, because reserving a condition only works if you captured enough of it to evaluate on and never trained against it. This is the same discipline behind what cannot be reconstructed once the rig comes down, applied to diversity rather than to calibration.
Annotation can be added later. Labels can be corrected, language descriptions written, and formats converted at any point from a faithful recording.
Coverage cannot. If your dataset contains four buildings, no downstream process produces a fifth. Camera intrinsics, mounting geometry, and lighting conditions are properties of the moment of capture, and an unrecorded participant identifier cannot be reverse-engineered from video. Skew discovered during evaluation can only be fixed by collecting again, which is why the axis targets belong in the protocol rather than in the analysis plan. Anyone weighing where to source this data is really choosing which coverage they will be able to obtain, and that decision is upstream of everything else, as the data problem inside embodied systems usually turns out to be.
Coverage is a protocol decision, and protocol decisions are cheapest before the first session. HumanSignal Services designs egocentric collection against stated per-axis quotas, runs the operation across participants and sites, and delivers episodes with condition metadata recorded rather than inferred. See how household capture is run to a quota, or book a scoping conversation to work through the axes your deployment implies.
It depends on whether participant variation is a nuisance to average out or a property the model must handle, and the answer differs by task. For manipulation policies sensitive to hand geometry and reach, tens of participants is a reasonable floor and single-digit counts usually bake in one person's body. Ego4D used 923 participants for broad activity understanding, which is far beyond what a focused manipulation dataset requires.
Not to the captured content. You can add annotation, correct labels, and reformat, but you cannot add a building, a lighting condition, or a participant that was never recorded. The practical move is to measure coverage per axis on what you have, then run a targeted supplementary collection aimed only at the thin cells.
It helps generalization across rigs and costs you per-rig density, so it is a real trade rather than a free gain. If deployment will use one known device, matching it is more valuable than spreading across several. If the deployment hardware is undecided or expected to change, capturing at least two configurations avoids learning one viewpoint as though it were the task.
Set a cap on hours per building and per lighting condition, then track coverage during collection rather than at the end. Convenience sampling reliably concentrates on well-lit, tidy, easily booked spaces, which is the opposite of deployment. Recruiting explicitly for cluttered and poorly lit spaces usually requires paying more for those sessions.
Yes, if the model will encounter them, and clean capture is the more common mistake. Policies trained only on tidy successful sequences have no representation of occlusion, recovery, or interruption. Capture failure with the cause annotated rather than discarding the episode, so it can be used for recognition and recovery training.
At minimum: participant identifier, environment identifier, task class, rig configuration with camera intrinsics and mounting geometry, and condition flags for lighting and clutter. Add measured per-frame timing rather than a declared frame rate. These are the fields that make coverage measurable, and every one of them is unrecoverable once the session ends.