Part 4 of our series on the data problem in Vision-Language-Action research. [Part 1: The Bottleneck Overview · Part 2: The Language-annotation Gap · Part 3: The Failure Data You're Throwing Away]
You fine-tune a Vision-Language-Action model, run it on LIBERO, and post a 95% success rate. The number goes in the paper, the demo video, the investor update. Then you deploy the same model on a real robot in a slightly different room, and it faceplants.
This isn’t uncommon, and the gap is not just bad luck. Across the 1,228 VLA papers we analyzed from arXiv cs.RO, a wave of diagnostic work published in 2025 and 2026 all arrived at the same verdict: the benchmarks the field lives by are good at measuring memorization, not capability, and the headline numbers are inflated by construction. This post walks through the evidence, explains why it happens, and describes what honest evaluation data looks like.
LIBERO is the universal evaluation environment for manipulation VLAs. By our count it appears in more than 300 of the 1,228 papers we read: if you report VLA results, you almost certainly report LIBERO. That ubiquity is exactly what makes its fragility a field-wide problem.
Two 2025 diagnostic studies stress-tested it, and the results are hard to unsee.
LIBERO-Plus (arXiv:2510.13626) applied modest perturbations to the standard tasks: small changes to lighting, camera angle, object placement, the kind of variation any real deployment produces on day one. Success rates fell from roughly 95% to below 30%. The models had learned the benchmark, not the task.
LIBERO-PRO (arXiv:2510.03827) went further and found the failure mode is more total than a degradation curve suggests. Under its perturbations, models posting 90%-plus success collapsed to 0.0%. The authors attribute the gap to rote memorization of action sequences and environment layouts. The same study, which we covered in Part 2, showed the models ignoring their language instructions entirely, executing the memorized sequence regardless of what they were told. Memorization and language-blindness are the same pathology seen from two angles.
This is not a mysterious result. It is what your training and evaluation setup optimizes for.
Mechanistic evidence makes it concrete. A sparse-autoencoder analysis of VLA internals (arXiv:2603.19183) found that the majority of features learned by models fine-tuned on small datasets correspond to memorized training sequences. The model is not building a reusable representation of "pick up the mug"; it is storing "in this layout, emit this trajectory." A benchmark that reuses its layouts at test time cannot tell the difference between a model that understands the task and one that has memorized the answer key.
Connect this to Part 3 and the picture closes. Success-only training gives the model no concept of recovery; memorization-rewarding benchmarks give it no reason to generalize; language-poor data gives it no reason to listen. Three findings, one brittle robot that aces the test and fails the room.
Set aside the perturbation problem and there is still a measurement problem, because the field's headline metric hides most of what happens during a rollout.
Binary success rate collapses an entire attempt into pass or fail. MetaFine (arXiv:2605.19986) argues that this collapse inflates reported performance by up to 70% relative to a graded assessment of what the policy actually accomplished. A model that grasps the right object, moves it most of the way, and drops it at the edge of the goal scores identically to one that never moved, and the binary number rewards near-misses and lucky completions alike.
Sample size compounds it. Real-world VLA evaluations typically rest on 25 or fewer rollouts reported without confidence intervals (arXiv:2605.29710). At that sample size, the difference between a 72% and an 80% model is inside the noise, yet papers routinely claim exactly those comparisons. You cannot resolve an eight-point difference from 25 trials, and most of the field's reported rankings sit inside their own error bars.
There is a security dimension to uncurated benchmark and training data that the field is only beginning to reckon with. Two 2025 papers demonstrated data-poisoning attacks on VLA training; one achieved 98–99% backdoor success by corrupting just 0.31% of training episodes (arXiv:2510.10932). If your evaluation set can't detect a policy that has memorized its way to a high score, it certainly can't detect one that has been quietly compromised. Trustworthy evaluation is a prerequisite for trustworthy deployment, not a reporting formality.
The diagnostic papers do not just complain; they point at the fix. Honest evaluation for VLAs has four properties, and each one is a data-construction problem before it is a metrics problem.
Deliberately perturbed, held-out conditions. An evaluation set has to contain variation the model never saw in training: new lighting, new camera poses, new object arrangements, new backgrounds, paraphrased instructions. LIBERO-Plus and LIBERO-PRO are useful precisely because they build this perturbation in. The moment your test conditions overlap your training conditions, you are measuring recall.
Graded outcomes, not binary flags. Because binary scoring inflates by up to 70%, an eval set needs partial-progress structure: did the policy reach the object, achieve the grasp, complete the transport, satisfy the goal. Eval-Actions (arXiv:2601.18723) is built along these lines, pairing scenarios with expert grading, ranked preferences, and reasoning traces, and it deliberately includes failure scenarios alongside successes. That grading is human judgment applied to rollouts, one clip at a time.
Adequate, reported sample sizes. Twenty-five rollouts without intervals cannot support the claims made on top of them. Honest evaluation means enough trials to compute confidence intervals and enough discipline to report them, which in turn means enough labeled ground truth to score against.
Human verification in the loop. Every property above rests on judgment a generic auto-grader can't supply. Deciding whether a rollout "recovered" or "half-succeeded," whether a perturbation is realistic, whether two attempts are genuinely comparable, is exactly the physical-causation reasoning that 2D-pretrained VLMs demonstrably lack (arXiv:2604.18484), the same limitation we traced through Parts 2 and 3. The evaluation set that catches a memorizing model is built by people who can tell memorization from understanding.
If you evaluate VLA models, the literature suggests a short, concrete checklist.
First, perturb your eval set and re-run. Change lighting, camera pose, and object placement on your existing benchmark and watch the number. If 95% becomes 30%, you have been measuring memorization, and now you know your real operating point.
Second, grade, don't flag. Score partial progress on your rollouts so a near-miss and a no-op stop looking identical. The 70% inflation in binary scoring is the difference between the model you think you have and the one you'll deploy.
Third, count your rollouts and report intervals. If a claim rests on 25 trials, treat it as a hypothesis, not a result.
Fourth, build a held-out, human-verified evaluation set as a deliverable in its own right, not an afterthought scraped from training leftovers. It is the instrument every other decision depends on, and right now the field is flying on a broken one.
A benchmark score is a measurement, and a measurement you can game is worse than no measurement, because it tells you that you're done when you aren't. The field spent three years optimizing against evaluations that reward memorization so the labs that build honest eval sets first will be the ones who find out what their models actually do before their customers do.
Next in the series: synthetic data is winning, and why that makes a trust layer more important, not less.
This post draws on our analysis of 1,228 VLA papers from arXiv cs.RO (Feb 2023 – Jun 2026). See Part 1 for methodology.