Part 3 of our series on the data problem in Vision-Language-Action research. [Part 1: the bottleneck overview · Part 2: the language-annotation gap]
Every robot data collection session produces two datasets, and you keep only one.
Watch any teleoperation collection effort: the operator attempts a task, the gripper slips, the mug tips over, the operator marks the episode failed and discards it. The session continues until you bank enough successful demonstrations, and those become the training set. The failures, often a substantial fraction of everything recorded, go to /dev/null as noise.
Across the 1,228 Vision-Language-Action papers we analyzed, this practice shows up as one of the field's most consequential mistakes. As one 2025 paper puts it, VLA models train on teleoperated successful demonstrations while the failed attempts that occur during collection get discarded, yet those failures encode where and how policies are fragile (arXiv:2512.03913).
This post covers why success-only training produces brittle robots, what the new generation of failure-centric datasets looks like, and what a failure-annotation schema contains, for teams that want to stop deleting half their data.
The core problem is an old one in imitation learning, now playing out at VLA scale: a model trained on success has never seen trouble, so it can't recognize trouble or escape from it.
Behavioral cloning teaches your policy to stay on the manifold of expert behavior. Execution is never perfect. A mistimed grasp, a localization error, an object that shifts mid-reach, and your robot sits in a state that appears nowhere in its training data. With no corrective signal, the policy's errors compound: as one analysis puts it, success-only behavioral cloning leaves models brittle, and minor execution errors cascade into unrecoverable, out-of-distribution failures (arXiv:2605.08434).
The downstream gap is structural. Existing manipulation datasets provide ground-truth trajectories only, leaving your robot with no model of what to do once something goes wrong (arXiv:2510.01642). The few datasets that address failure offer textual explanations of what went wrong: useful for analysis, but a VLA model can't consume them as actions (arXiv:2510.01642).
So the field has built robots that perform until the first perturbation, then have no concept of recovery. If you read Part 1 of this series, that should sound familiar; it's the same brittleness the benchmark-fragility studies exposed, approached from the data side. Success-only data and memorization-rewarding benchmarks form two halves of one problem.
The clearest evidence that failure data matters is the cluster of 2025–2026 releases built to supply it. Each one was publishable as a dataset because nothing like it existed.
RoboFAC (arXiv:2505.12224): 9,440 erroneous manipulation trajectories paired with 78,623 question-answer pairs, with failures categorized by type. The framing is "failure analysis and correction"; the dataset teaches models to diagnose what went wrong as well as act.
FailSafe (arXiv:2510.01642): the key advance is failures paired with executable recovery actions. This is the format a VLA can train on, with motor-level instructions for getting out of a given failure state.
ViFailback (arXiv:2512.02787): 58,126 failure-diagnosis VQA pairs plus visual-symbol correction annotations. The authors observe that prior failure datasets came from simulation, which limits real-world transfer; the argument is that real failure data, from real collection sessions, is the scarce and valuable kind.
SAFE (arXiv:2506.09937): a failure detector trained on both successful and failed rollouts, demonstrating that the simplest use of failure data (binary detection: is this episode going wrong?) requires examples of episodes going wrong.
Eval-Actions (arXiv:2601.18723): includes failure scenarios alongside expert grading, ranked preferences, and reasoning traces, treating failure as a first-class part of evaluation data.
On the corpus side, the flagship open datasets have started shipping failures as a feature: RoboMIND includes 5,000 labeled failure demonstrations with annotated causes (arXiv:2412.13877), and AIRoA MoMa builds failure cases into its annotation schema alongside force-torque signals (arXiv:2509.25032).
Beyond static datasets, the most interesting training results use failure as signal. VINE turns discarded failed attempts into learning signal (arXiv:2512.03913). GRAPE performs trajectory-level preference alignment using both successes and failures; the contrast between them is the supervision (arXiv:2411.19309). And VLAC trains a process reward model that lifted real-world task success from 30% to 90% within 200 episodes (arXiv:2509.15937), a result that depends on graded supervision over partial progress and failure; binary labels can't carry that signal.
The pattern across the corpus: failure data is the missing half of the supervision signal, and the groups that have it are publishing the robustness results that your benchmarks will compare against.
Suppose you stop deleting failed episodes tomorrow. Raw failures are not yet training data; the value sits in the annotation layer. Synthesizing the schemas used by RoboFAC, FailSafe, ViFailback, RoboMIND, and AIRoA MoMa, a complete failure annotation has four components, in increasing order of value and annotation effort.
1. Detection: when did it go wrong? A timestamp or frame index marking the failure point, separating the healthy prefix of the trajectory from the failed suffix. This minimal label powers failure detectors like SAFE and lets you salvage the successful prefix as training data.
2. Categorization: what kind of failure? A taxonomy label: grasp slip, collision, wrong object, dropped object, kinematic limit, perception error, instruction misunderstanding. RoboFAC's categorized failure types and RoboMIND's annotated causes both take this form. The taxonomy matters more than its granularity; a consistent twelve-category scheme applied across your data beats an exquisite ontology applied ad hoc, because the downstream use is aggregate. It tells you where your policy is fragile, which redirects both training emphasis and future collection.
3. Causal explanation: why did it fail? A natural-language or structured account linking the failure to its cause: "the grasp approached from above but the handle was oriented laterally." This is what RoboFAC's 78K QA pairs and ViFailback's 58K diagnosis pairs encode, and it's what produces failure-reasoning models past simple detectors. Note the connection to Part 2: causal failure explanation is the kind of dense, diverse, fine-grained language annotation that the corpus shows your models are starving for.
4. Recovery: what should happen next? The highest-value layer and the rarest: an executable recovery action or demonstrated recovery trajectory from the failure state, FailSafe-style. In live collection, this can run close to free. When a teleoperator's attempt slips and they recover and complete the task, that recording is a failure-plus-recovery pair, if your pipeline segments and labels it instead of discarding the blemished episode.
A fifth element rewards designing in from the start: preference pairs. A failed and a successful attempt at the same task, from the same initial conditions, gives you a preference pair for GRAPE-style alignment or reward-model training. Your collection sessions generate these pairs in every shift; few pipelines retain them.
If you're building a VLA data pipeline, much of it now runs through VLM-based labelers, and Part 2 covered where that works and where it doesn't. Failure annotation sits on the "doesn't" side for a structural reason: steps 2 through 4 require judgment about physical causation. Deciding why a grasp failed (bad approach angle, insufficient force, object slippage) requires reasoning about contact, geometry, and intent that generic VLMs, trained on 2D image-text data, lack (arXiv:2604.18484). The recovery layer requires knowing what a competent operator would have done, which makes your operators' judgment irreplaceable.
That's also why failure data is structurally scarce in a way that success data is not. Synthetic data engines can generate successful trajectories by the billion, but they train on demonstration data that lacks failure coverage (arXiv:2602.12063), and simulation-generated failures fail real-world transfer (arXiv:2512.02787). The valuable failure data comes from one place: real collection sessions, annotated by people who understand what they're looking at.
For any team running robot data collection, the changes are concrete and cheap.
Stop deleting failed episodes; storage is not your bottleneck. Mark failure timestamps at collection time, when the operator's memory is fresh and the label costs seconds. Adopt a small, consistent failure taxonomy and apply it to everything. When operators recover mid-episode, keep the whole episode and segment it; that's a failure-recovery pair, the rarest data type in the field. And retain matched success/failure attempts as preference pairs.
The field's own numbers suggest what this is worth: process-reward supervision built on graded success and failure took one system from 30% to 90% real-world success in 200 episodes (arXiv:2509.15937). Your operators generate the raw material for that kind of result in every collection session, every day, and then throw it away.
Next in the series: why a 95% benchmark score can collapse to zero, and what honest evaluation data looks like.
This post draws on our analysis of 1,228 VLA papers from arXiv cs.RO (Feb 2023 – Jun 2026). See Part 1 for methodology.