A million trajectories, 85% from four robots

TL;DR. The Open X-Embodiment collaboration pooled 60 robot datasets across 22 embodiments, covering 527 skills and 160,266 tasks, and the resulting policies showed real gains: RT-1-X achieved a 50% higher success rate than the specialist methods it was compared against, and RT-2-X showed roughly threefold generalisation improvement on skills absent from the evaluation robot's own training data. Positive transfer across robot bodies is now demonstrated, which is a genuine result. And the dataset it rests on has over 85% of its real trajectories from just four robot arms, with the largest generalist policies trained on around a million trajectories. Language models train on trillions of tokens. Text was written anyway. Robot trajectories have to be manufactured, one physical motion at a time, which is why redefinition dominates deployment and will keep doing so.

---

Status: established. Primary source: Open X-Embodiment, arXiv:2310.08864, ICRA 2024, a collaboration whose own discussion section reports 22 embodiments from 21 institutions; secondary coverage cites larger participation figures and the paper's own number is used here. The imbalance figure is from subsequent analysis of the dataset.

---

What the field achieved

Worth stating properly, because the result is real and often reported as though it were not.

Open X-Embodiment consolidated 60 existing robot datasets spanning 22 different robot bodies, producing a corpus of 527 skills across 160,266 tasks.

Policies trained on the pooled data outperformed the specialists. RT-1-X reached a 50% higher mean success rate than the state-of-the-art methods each contributing institution had built for its own robot. RT-2-X, built on a vision-language model, showed approximately threefold improvement on emergent skills, meaning tasks not present in the evaluation robot's own training data.

That is the central hypothesis confirmed: diverse experience across different robot bodies produces better policies than specialist training on one. The paradigm that worked for language and vision transfers to robotics in kind, if not yet in degree.

Evaluation was disciplined too: fixed sets of five to six skills per robot, 100 trials per skill, binary success, with held-out objects, unseen backgrounds and novel language commands for the out-of-distribution tests.

And what it rests on

Over 85% of the real trajectories come from four robot arms: Franka, xArm, Kuka iiwa, and Google's research robot. Many of the 22 embodiments appear in only one or two constituent datasets.

Subsequent analysis notes that most component datasets are tied to a single robot in a fixed environment, which risks policies learning robot-and-scene combinations rather than tasks.

The largest generalist policies are trained on roughly a million trajectories.

The number that explains this territory

A million trajectories.

Frontier language models train on trillions of tokens. That is six orders of magnitude, and the gap is not a funding problem.

Text existed already. Every book, comment, manual and article was written by people for their own reasons, and the model got it as a byproduct. Images existed already. Photographs were taken because someone wanted a photograph.

A robot trajectory has never existed until a robot performs it.

Each one requires a physical machine to physically move through a real task, in real time, once. A ten-second manipulation takes ten seconds. It cannot be parallelised across a dataset that is already written, because there is no such dataset. It has to be generated, on hardware, per attempt, and the hardware costs thousands to hundreds of thousands of dollars per unit.

This is the structural asymmetry underneath every article in this territory. Industrial robots engineer the workspace, vehicles narrow the domain, crops get bred for machines, drones delete the landing, and vacuums widen the tolerance. All five are ways of getting a useful system without the data that would make redefinition unnecessary.

What is being tried

Four approaches, and they are the ones to watch.

Pooling. Open X-Embodiment itself: combine what exists and exploit transfer. Demonstrated to work, and bounded by what exists.

Simulation. Generate trajectories in physics engines where a ten-second task can run in milliseconds and in parallel. Bounded by the sim-to-real gap, where policies meet dynamics, sensor noise and material properties the simulator approximated.

Human video. Learn from recordings of people doing tasks, which does exist at scale. Bounded by the absence of action labels: video shows what happened, not what forces were applied or what the wrist did.

And pretrained vision-language backbones. Start from a model that already understands objects and instructions, and learn only the action mapping. This is what RT-2-X does and it is why its generalisation gain was the larger of the two.

Each converts a data problem into a different problem. None of them produces internet-scale robot experience, because that is not a thing that can be produced by scraping.

Three things this establishes

Positive transfer across robot bodies is demonstrated, not speculative. A 50% success improvement and threefold emergent-skill gain against specialist baselines is a real finding, and anyone claiming robot learning does not scale has to account for it.

The data ceiling is physical, not financial. More money buys more robots collecting more trajectories in real time. It does not buy a corpus that already exists, because the corpus was never written.

And the imbalance matters for what the results mean. With 85% of trajectories from four arms, a demonstrated generalisation gain is generalisation across a population dominated by four platforms in laboratory settings. That is construct validity applied to a dataset: the result is evidence about the corpus that produced it.

What it does not establish

That the data problem is unsolvable. Simulation, human video and cross-embodiment transfer are all active and improving, and the field is a few years into serious effort.

That scale is the only missing thing. It may be that manipulation needs architectural or representational advances that data alone will not supply, and that is an open question rather than a settled one.

That the trillion-token comparison is apt. Tokens and trajectories are not commensurable units, and a trajectory carries far more information than a token. The six-order-of-magnitude figure is a scale contrast, not an equivalence.

And nothing about timelines. This article makes no prediction about when manipulation is solved, because the honest answer is that nobody knows and the people closest to it disagree.

What is unresolved

Whether simulation closes the gap. The bet is substantial and the evidence is mixed. Policies trained in simulation transfer better than they used to and not reliably.

What human video can supply without action labels. Recovering forces and joint states from video is an active problem, and how much of manipulation is recoverable that way is unknown.

Whether the four-arm concentration corrects. New datasets are being collected on more diverse hardware, and whether the resulting corpus is balanced enough to support genuine cross-embodiment claims is not yet clear.

And whether a data-collection flywheel exists. Deployed robots could in principle collect trajectories continuously, which is how the asymmetry would eventually close. Nobody has published evidence of one operating at meaningful scale.

The counter-argument

The comparison to language data is misleading. A robot trajectory contains vastly more information than a token: proprioception, force, vision, and a temporally extended action sequence. A million well-chosen trajectories may carry more usable signal than a trillion tokens of forum posts, and counting units across modalities is close to meaningless.

Data scarcity may not be the binding constraint. Human infants learn manipulation from far fewer than a million examples, which suggests the sample-efficiency of current methods is the problem rather than the size of the corpus. If so, more data is the expensive route to a solution that a better method would reach cheaply.

The imbalance criticism cuts less than it appears. Four arms is few, and the demonstrated transfer was to embodiments outside the dominant four, which is the harder direction. A concentrated training distribution producing gains on underrepresented platforms is evidence for transfer rather than against it.

And redefinition is not a consolation prize. Framing the five strategies as workarounds for missing data implies the goal is a robot that needs no accommodation. That goal is a research aspiration; the accommodations are how every deployed system works, including the ones that work extremely well.

The short version

Open X-Embodiment pooled 60 robot datasets across 22 embodiments, covering 527 skills and 160,266 tasks. Policies trained on it beat the specialists: RT-1-X by 50% mean success, and RT-2-X by roughly threefold on emergent skills absent from the evaluation robot's own data. Positive transfer across robot bodies is a demonstrated result, not a hope.

And over 85% of the real trajectories come from four robot arms, with the largest generalist policies trained on about a million trajectories.

Frontier language models train on trillions of tokens. That is six orders of magnitude, and it is not a funding gap.

Text existed already. So did images. Every one was produced by a person for their own reasons and the model received it as a byproduct. A robot trajectory has never existed until a robot performs it, on hardware, in real time, once. A ten-second manipulation takes ten seconds.

That asymmetry is the reason this whole territory looks the way it does. Workspaces get engineered, domains get narrowed, crops get bred, landings get deleted, tolerances get widened. All five are ways of building something useful without the data that would make accommodation unnecessary.

Four routes are being tried: pooling what exists, generating trajectories in simulation, learning from human video without action labels, and starting from pretrained vision-language models. Each converts the data problem into a different problem, and none produces internet-scale robot experience, because that is not something scraping can produce.

Common questions

What is Open X-Embodiment? A collaboration that consolidated 60 existing robot datasets spanning 22 different robot bodies into a single corpus covering 527 skills and 160,266 tasks, published as arXiv:2310.08864 and presented at ICRA 2024. Its purpose was to test whether the large-and-diverse-data paradigm that worked for language and vision also works in robotics.

Did it work? Yes, measurably. RT-1-X achieved a 50% higher mean success rate than the state-of-the-art specialist methods each contributing institution had built for its own robot. RT-2-X, built on a vision-language model, showed roughly threefold improvement on emergent skills, meaning tasks absent from the evaluation robot's own training data. Evaluation used fixed skill sets with 100 trials each and held-out objects, backgrounds and language for the out-of-distribution tests.

What is the limitation? Over 85% of real trajectories in the pooled dataset come from four robot arms: Franka, xArm, Kuka iiwa, and Google's research robot. Many of the 22 embodiments appear in only one or two constituent datasets, and most component datasets are tied to a single robot in a fixed environment, which risks policies learning robot-and-scene combinations rather than tasks.

How does the data scale compare to language models? The largest generalist robot policies are trained on around a million trajectories. Frontier language models train on trillions of tokens, a difference of about six orders of magnitude. The units are not commensurable, so this is a scale contrast rather than an equivalence, and a trajectory carries far more information than a token.

Why can't robot data be scraped like text? Because it does not exist until it is made. Every book, comment and manual was written by a person for their own reasons, and a language model receives it as a byproduct of human activity. A robot trajectory requires a physical machine to physically move through a task in real time, once. A ten-second manipulation takes ten seconds and cannot be parallelised across a corpus that was never written, and the hardware costs thousands to hundreds of thousands of dollars per unit.

How does this relate to the rest of this territory? It is the explanation underneath it. Industrial robots engineer the workspace, autonomous vehicles narrow the operating domain, row crops were bred for machines, delivery drones delete the landing, and robot vacuums widen the tolerance. All five are ways of getting a useful system without the data that would make accommodation unnecessary. The accommodations are not a failure of ambition; they are what is available given the corpus that exists.

What approaches might close the gap? Four. Pooling existing data and exploiting cross-embodiment transfer, which is demonstrated and bounded by what exists. Generating trajectories in simulation, where a ten-second task runs in milliseconds and in parallel, bounded by the sim-to-real gap. Learning from human video, which exists at scale but lacks action labels showing what forces were applied. And starting from pretrained vision-language models so only the action mapping must be learned, which is what produced the larger of the two reported gains.

Is data scarcity definitely the binding constraint? No, and the strongest counter-argument is that human infants learn manipulation from far fewer than a million examples, which points at the sample efficiency of current methods rather than the size of the corpus. If that is right, more data is the expensive path to something a better method would reach cheaply. Whether manipulation needs more data, better methods, or both is genuinely open, and the people closest to it disagree.