Part 2 of our series on the data problem in Vision-Language-Action research. [Part 1: We Read 1,228 VLA Papers. The Bottleneck Isn't What You Think.]
Take a state-of-the-art Vision-Language-Action model, one posting 90%+ success rates on a standard benchmark, and corrupt its language input. Swap "put the mug in the sink" for a paraphrase. Or for an instruction about a different object. Or for meaningless tokens.
In a 2025 study, researchers ran this experiment and the model's outputs didn't change (arXiv:2510.03827). It executed the same action sequence regardless of what it was told. A companion diagnostic study found that under perturbation, models "tend to ignore language instructions completely" (arXiv:2510.13626). A third team, building a benchmark to measure language grounding, found that current state-of-the-art VLA models disregard their instructions (arXiv:2603.00592). In many deployed models, the L in VLA does no work.
This post explains the data cause behind that behavior, why architectural changes fall short, and a fix that is cheap, quantified, and doesn't require collecting a single new trajectory.
When researchers audit the language annotations in today's robot-learning corpora, they find the same thing. A systematic linguistic analysis of embodied-AI datasets (arXiv:2601.03136), the most direct study on this question, found that most datasets rely on template-like commands with minimal structural variation. The instructions read like a script generated them, because a script did: "pick up the {object}," "put the {object} in the {container}," repeated thousands of times with the slots filled in.
Compare that to the other two modalities in your training data. The visual stream is rich: every demonstration has different lighting, object positions, backgrounds, occlusions. The action stream is rich: every trajectory is a unique sequence of continuous motor commands. The language stream is a handful of templates.
The authors of a 2025 paper gave this asymmetry a name: modality imbalance. Language diversity in VLA datasets sits an order of magnitude below visual and action diversity, and the imbalance biases models toward visual shortcuts and what the authors call language forgetting (arXiv:2512.11218).
Why does low language diversity cause your model to ignore language?
Consider what your model is optimizing. It learns to predict actions from two inputs: what it sees and what it's told. If the instruction is predictable from the image, and in goal-driven datasets the scene tends to predict the instruction (one mug, one sink, "put the mug in the sink"), then language carries no information your model can't extract from the image. The conditional mutual information between instruction and action, given the image, approaches zero. The authors term this Information Collapse (arXiv:2601.15197): the dataset's own construction makes language redundant, so a well-optimized model discards it.
A parallel paper describes the same phenomenon as modality collapse: strong visual priors overwhelm sparse linguistic signal, and your agent overfits to specific instruction phrasings while losing the underlying semantic intent (arXiv:2601.04052). Your model optimizes for what your data measures, and your data measures visual context.
There's a subtler version of the problem when your model does attend to language. The authors of FineVLA (arXiv:2605.27284) note that existing datasets pair trajectories with coarse, goal-level language while leaving execution-critical details unspecified: which arm to use, which direction to approach from, where to make contact. They call this supervision aliasing: many distinct correct behaviors collapse onto one vague label, so you can't steer the model at the level that matters.
The consequences appear the moment your model leaves its training distribution.
The benchmark evidence is stark. LIBERO-PRO showed 90%+ success rates collapsing under instruction perturbation, with outputs invariant even to corrupted or meaningless instructions (arXiv:2510.03827). A controlled robustness study found that paraphrasing instructions, same task in different words, degrades performance by 22 to 52 percentage points (arXiv:2603.28301). That is the gap between a demo and a product: your users do not speak in your dataset's templates.
The problem compounds with the memorization issue we covered in Part 1. A model that ignores language has one strategy left: memorize which action sequence goes with which visual layout. Sparse-autoencoder analysis confirms that fine-tuned VLAs do this (arXiv:2603.19183).
If you are training or fine-tuning a VLA model, this is the most actionable finding in the entire 1,228-paper corpus. Multiple groups have tested the obvious remedy, relabeling existing trajectories with diverse, fine-grained language, and the gains are large.
LangGap (arXiv:2603.00592) built a benchmark to isolate language grounding, confirmed that SOTA models ignore instructions, then applied targeted instruction augmentation: re-annotating the same trajectories with varied, discriminative language. Single-task success went from 0% to 90% on the same trajectories with the same architecture.
CAST (arXiv:2508.13446) took a counterfactual approach for navigation. The authors relabeled existing trajectories with alternative instructions the trajectory also satisfies, plus instructions it doesn't, forcing the model to attend to the distinction. Result: +27 percentage points, achieved, in the authors' words, without any additional data collection.
FineVLA (arXiv:2605.27284) attacked supervision aliasing by relabeling 47,000 trajectories (selected from 972,000) with fine-grained, execution-level language: active arm, approach direction, contact region. The team used a robotics-specialized annotator with humans verifying in the loop. The result is a model your team can steer.
SteerVLA (arXiv:2602.08440) found fine-grained language annotation "essential" for reasoning and steerability. The most rigorous controlled comparison available, a large co-training study on language-conditioned behavior models (arXiv:2602.01067), tested which auxiliary data modalities help and found dense language annotations on trajectories among the few interventions with clear positive effect.
Consider the economics. Collecting one hour of new teleoperated robot data requires a robot, an operator, a rig, and a lab. Relabeling an hour of existing data with diverse, fine-grained language requires a person and an annotation interface. The corpus's own numbers say the second intervention can be worth 27 to 90 points of task success. No other known intervention in robot learning has that ratio of cost to measured impact.
The tempting shortcut: point a large vision-language model at your dataset and have it generate diverse annotations. Some of it works; FineVLA's pipeline uses a specialized VLM annotator. The corpus carries three warnings.
First, generic auto-labelers reproduce the template bias you're trying to escape. An LLM asked to caption ten thousand pick-and-place clips converges on its own templates. The diversity that breaks Information Collapse requires deliberate, human-quality variation: paraphrase styles, referential strategies ("the red one," "the mug nearest you," "the one you just moved"), negations, and counterfactuals.
Second, auto-labelers produce silent failures. Synthetic-data pipelines now build in VLM critics to filter their own output (arXiv:2604.09036), and the critics themselves need verified ground truth to calibrate against.
Third, the fine-grained spatial details that fix supervision aliasing, contact regions and approach directions, are where generic VLMs are weakest. Their 2D image-text pretraining never taught them the geometric reasoning the task requires (arXiv:2604.18484). FineVLA's answer was human verification in the loop; that's the pattern that works.
If you're training or fine-tuning a VLA model, the literature suggests a concrete sequence.
First, audit your language. Count unique instruction structures, not unique strings. If your dataset's language reduces to a few dozen templates, and it likely does, you have modality imbalance.
Second, test whether your model is using language at all. The diagnostic is free: paraphrase your eval instructions and watch what happens. A 20+ point drop means your model passes benchmarks on memorization.
Third, relabel before you collect. Diverse paraphrases, discriminative and counterfactual instructions, and fine-grained execution-level detail on the trajectories you already own. The measured ceiling on this intervention, 0% to 90%, is higher than what you'd buy with the same budget in new data collection.
The field spent three years assuming the language half of vision-language-action came for free from internet-pretrained backbones. The evidence says otherwise: your team builds or breaks language grounding in the robot dataset itself, one annotation at a time.
Next in the series: the other half of every data collection session, the failures everyone deletes, and why they're worth keeping.
This post draws on our analysis of 1,228 VLA papers from arXiv cs.RO (Feb 2023 – Jun 2026). See Part 1 for methodology.