Part 5 of our series on the data problem in Vision-Language-Action research. [Part 1: the bottleneck overview · Part 2: the language gap · Part 3: the failure data · Part 4: the evaluation crisis]
If you've read the first four parts of this series, you might expect this one to argue against synthetic data. It doesn't. The synthetic-data boom in robotics is real, it's accelerating, and on the evidence of the 1,228 VLA papers we analyzed, it is one of the most promising answers to the data bottleneck the field faces. Consequently, fighting it would be at worst, a mistake and at best, a bit of a waste of time.
The argument here is narrower and, we think, more useful: synthetic data scales volume faster than it scales trust, and the gap between the two is exactly where curation, verification, and human-labeled ground truth earn their keep. Give synthetic its due, then look hard at the three limits outlined by the corpus documents and backed by our experience.
Skepticism about simulated and generated robot data is out of date. Recent publications on the topic contain results that would have sounded implausible two years ago.
The clearest is InternData-A1 (arXiv:2511.16651), which claims the first synthetic-only corpus to train a VLA policy that matches one trained on a strong real-robot dataset. That is the milestone the field has been chasing: not "synthetic helps as augmentation," but "synthetic alone is competitive." Alongside it, data engines like GigaBrain-0 (arXiv:2510.19430) argue directly from the economics: physical data collection is too inefficient to scale VLA systems, so generation has to carry more of the load. Generation pipelines such as GigaWorld-0 and large-scale grasp corpora like SynGrasp-1B are pushing sheer volume far beyond what teleoperation can reach.
The appeal is obvious. Synthetic data is cheap per episode, infinitely variable, perfectly labeled at the point of generation, and free of the operator-scheduling and hardware constraints that make real collection so painful. If you need a billion grasps, you are not going to teleoperate them.
So the question is not whether to use synthetic data. You should. The question is what breaks when you trust it uncritically, and the research is specific about that.
Matching real data on an in-distribution benchmark is not the same as working in the world. The corpus is full of policies that look excellent in simulation and evaporate on hardware: in one reported comparison, a policy scoring 97.65% on LIBERO dropped to essentially 0% on a real-world manipulation task. The number that mattered was the second one.
This is the same lesson as [Part 4] (link), arriving from a different direction. There, the gap was between a memorized benchmark and a perturbed one; here, it's between a generated distribution and a physical one. Synthetic data can inadvertently widen it, because a generator reproduces the visual and physical statistics it was built with, and those statistics are never quite real. You need real, held-out, human-verified evaluation to know whether your synthetic-trained policy actually transfers, and that evaluation set cannot itself be synthetic without begging the question.
[In our earlier article] (link) we argued that the field throws away its failure data. Synthetic generation essentially launders it.
A generator trained on demonstration data that systematically lacks failure coverage will produce synthetic data that also lacks failure coverage (arXiv:2602.12063). It can render ten thousand pristine, successful pick-and-places, but it has never seen a grasp slip in a physically faithful way, so it can't teach a policy what slipping looks like or how to recover. The synthetic corpus inherits the success-only bias of its source and amplifies it, because now the bias comes with the credibility of volume.
The valuable failure data comes from real collection sessions annotated by people who understand physical causation. Synthetic generation makes that human-sourced failure data more scarce relative to the flood of clean synthetic success, not less important.
The subtlest problem is that generators fail quietly. A rendered episode can contain a physically impossible contact, an object that clips through a surface, a label that doesn't match the pixels, and nothing flags it. These silent failures poison training precisely because they look fine.
The field's own strongest pipelines concede the point by building verification in. The most capable synthetic engines now include VLM-based critics whose explicit job is to filter the generator's own output, discarding episodes that don't survive inspection (arXiv:2604.09036). This is an important tell: the people closest to synthetic generation do not trust it raw. They put a verifier between the generator and the training set.
And that verifier has a dependency of its own. A VLM critic is only as good as the ground truth it was calibrated against. To trust the critic, you need a human-verified reference set, the thing that tells you the critic is catching real problems and not inventing them. The verification stack bottoms out in human judgment; it just moves it one layer up.
There's an angle here that should matter to anyone deploying VLAs in a product, and it reframes curation from a quality nicety into a security control.
Two 2025 papers demonstrated data-poisoning attacks on VLA training. One achieved 98–99% backdoor success by corrupting just 0.31% of training episodes (arXiv:2510.10932). Now put that next to a pipeline ingesting millions of machine-generated episodes with minimal human review. A synthetic firehose is an ideal delivery mechanism for a poisoning attack precisely because no human looks at most of it, and 0.31% is a rounding error in a corpus that size.
Uncurated data, then, is not just a quality liability. It's an attack surface. The same curation and verification layer that filters silent generation failures is the layer that would catch a poisoned subset. Trust is not a soft property you add at the end; in an adversarial setting it's the difference between a model you can ship and a backdoor you can't see.
Put the three limits together and the shape of the solution is clear. As synthetic volume grows, the scarce and decisive asset becomes the layer that establishes whether the volume is any good. Concretely, a trust layer:
Maintains a real, human-verified evaluation set to measure sim-to-real transfer honestly, because a synthetic benchmark can't audit a synthetic generator. It supplies real, annotated failure data to cover the cases generators can't produce, the failure-plus-recovery pairs from Part 3 that no simulator has seen. It provides human-verified ground truth to calibrate the VLM critics that filter generated output, so the verifier is itself verified. And it applies human review to the curation pipeline as a security control, on the theory that a 0.31% poisoning threshold is too low to defend with automation alone.
None of this competes with synthetic generation. All of it makes synthetic generation safe to rely on. The labs winning with synthetic data are not the ones generating the most episodes; they're the ones who generate aggressively and verify ruthlessly, and who understand that the second half is where the trust, and the moat, actually lives.
Synthetic data is winning. Build the trust layer, and let it win for you instead of against you.
Next in the series: a million hours of human video, and why it's inert until someone labels it.
This post draws on our analysis of 1,228 VLA papers from arXiv cs.RO (Feb 2023 – Jun 2026). See Part 1 for methodology.