Machines That Move

Language models learned from text that already existed. Robots and autonomous vehicles have no such inheritance. Every example they learn from has to be made.

When researchers at Google set out to train a general-purpose robot controller, they needed examples of robots doing things. Nothing suitable existed. So they built the dataset: 13 robots, operated by people over 17 months in office kitchens, producing about 130,000 recorded episodes covering more than 700 tasks.

Compare that with the origins of a language model. Its training text was written over decades by billions of people for their own reasons, and was sitting on the internet waiting to be collected. The language-model developer’s problem was gathering and filtering. The robot developer’s problem is manufacturing.

That asymmetry explains much of why AI that writes has advanced faster than AI that acts.

There is no internet of actions

Text, images and video are by-products of human life, published in enormous quantities. The data a robot needs is different in kind. It needs records of what a body sensed, what it did in response, and what happened next: camera views from the robot’s own position, joint angles, forces, and the commands that produced them.

Nobody generates that as a by-product. It has to be recorded deliberately, on physical hardware, usually by a person operating the robot remotely to demonstrate the task.

The published figures give a sense of the cost.

DROID, a 2024 dataset built by a consortium of university and industry labs, took 50 data collectors working across three continents for 12 months to gather 76,000 demonstration trajectories, about 350 hours of robot interaction, across 564 scenes and 84 tasks. That is among the larger and more varied robot datasets in existence. It is 350 hours.

Open X-Embodiment, a collaboration of 21 institutions, pooled existing datasets from 22 different robot types into a common format, covering 527 skills. Its purpose was to test whether learning from many robots at once could help each of them, as learning from many languages helps a language model. The results suggested that it could. The exercise also showed how small and fragmented robot data is. A field-wide pooling of datasets from 21 institutions yielded a collection that would be unremarkable by the standards of text.

A language model may train on trillions of words. The leading robot datasets contain demonstrations numbered in the tens or hundreds of thousands, each of which took a person and a machine several minutes to produce.

Driving: plenty of data, not enough of the right kind

Autonomous vehicles look like the exception. Test fleets record continuously, and the leading developers have logged large distances. Raw sensor data is abundant.

The bottleneck there is of a different kind, and there are two parts to it.

Annotation is intensive. The public research datasets show what a labelled driving scene involves. nuScenes consists of 1,000 scenes of twenty seconds each, recorded with six cameras, five radars and a lidar unit. It contains 1.4 million camera images and 1.4 million three-dimensional bounding boxes across 23 object classes. The Waymo Open Dataset contains 1,150 scenes, also twenty seconds each, with around 12 million three-dimensional boxes.

Each box is a cuboid placed in a lidar point cloud, consistent with what the cameras show, carrying a class, a size, an orientation and an identity maintained across time. The annotator is working in three dimensions, across several sensors that have to be reconciled, tracking every object through the scene. An earlier dataset, Cityscapes, reported that annotating a single street image with fine pixel-level labels took more than an hour and a half on average. Annotating a multi-sensor sequence is a larger task.

The events that matter are rare. Almost all driving is uneventful. A child running into the road, debris falling from a truck, an unusual vehicle, a traffic officer overriding a signal: these are the cases a system must handle and the ones a fleet almost never records. The distribution of driving situations has a very long tail, and safety depends on performance in the tail.

So the industry’s practical problem is not obtaining data. It is finding, within vast recordings, the small fraction that is informative, and labelling it to a standard that supports safety claims. This is the active-learning setting described in an earlier article, at its most consequential.

What makes physical-world labels hard

Annotation for physical AI has features that do not arise in text or ordinary images.

Three dimensions. Labels describe position, extent and orientation in space. Errors of a few centimetres or degrees matter for a system that must not hit things.

Multiple sensors. Cameras, lidar and radar see differently and are mounted in different places. A label must be consistent across all of them, which depends on precise calibration and on annotators who understand what each sensor shows.

Time. Everything in the previous article about video applies. Objects must keep their identity. Their motion must be physically plausible.

Ambiguity of intent. Predicting what other road users will do, or whether a manipulation succeeded, requires judgement.

Success is not always visible. Did the robot place the object properly? Is the grasp stable? The answer may depend on forces and outcomes that a camera view does not settle.

Safety-critical evaluation. Labels used to validate a system’s safety need a higher standard of accuracy and auditability than labels used for training, for the reasons given in the article on test sets, with greater consequences.

The simulation argument

The natural response to scarce real data is to generate it. Simulators can produce unlimited driving scenes and robot trajectories with perfect labels, including dangerous situations that could never be staged.

Simulation is indispensable, and widely used for testing as well as training. Domain randomisation, described in the article on synthetic data, has allowed policies trained in simulation to work on real hardware for a growing range of tasks.

Its limit is the one set out in that article. A simulator contains what its builders put in. It reproduces the situations they thought to model, with physics as accurate as they made it. The unanticipated situation, which is the definition of the long tail, is absent from the simulator for the same reason it is absent from the engineers’ plans. Contact-rich manipulation, deformable objects, liquids and the behaviour of other people remain hard to simulate faithfully.

In practice, simulation and real data are used together. Real data grounds the simulator and reveals what it is missing. Simulation multiplies and varies what is known. Both need labelled real-world data as the reference.

The foundation-model argument

A second response is that large pretrained models will reduce the need for robot-specific data. Vision-language models trained on internet images and text already understand objects, scenes and instructions. Attach such a model to a robot and it may need only a little embodied data to learn to act.

There is evidence for this. Models that combine a pretrained vision-language backbone with robot data have shown better generalisation to new objects and instructions than models trained on robot data alone. Internet-scale pretraining does supply semantic knowledge: what a cup is, what “put it on the shelf” means.

It does not supply the mapping from that understanding to motor commands for a particular body, or the physical feel of contact, friction and weight. That part still has to come from embodied experience. The required amount may fall. The requirement does not go away, and as elsewhere in this series, less data that has to be more carefully chosen is a harder data problem, not an easier one.

What the data operation looks like

Producing data for physical AI is an industrial activity with several distinct parts.

Collection. Operating vehicles or tele-operating robots, with trained operators, in varied environments. Variety of scenes, objects and conditions matters more than hours in one setting, as the DROID authors emphasise.

Curation. Selecting the informative fraction from large recordings, using model uncertainty, disagreement between sensors or models, and triggers for rare events.

Annotation. Three-dimensional, multi-sensor, temporally consistent labelling, increasingly model-assisted, with people verifying and correcting.

Quality assurance. Geometric consistency checks, cross-sensor validation, and independent audit of safety-relevant labels.

Demonstration quality. For robot learning, the quality of the human demonstration is itself a property of the data. Hesitant or inconsistent demonstrations teach hesitant or inconsistent policies. Operator training and filtering of poor episodes are part of the pipeline.

Scenario description. Tagging recordings with structured descriptions of what happened, so that specific situations can be retrieved for training and testing.

What this changes