Polymorph/Physical AI, ExplainedGlossary

Physical AI, Explained · Part 4

Data, simulation and world models

Read this part in the book →

Learned robot behaviour is only as good as the data it is trained on, and robot data is scarce, expensive and uneven in quality. This part describes where robot data comes from, what each source is good for, how teams select and label it, and how simulation and world models stretch a limited supply of real experience further. It closes with the problem of measuring whether any of it worked.

4.1 Why robot data is hard to get

Language and vision models were trained on data that already existed: text and images accumulated on the internet over decades. No comparable archive exists for robots. A useful training example for a robot pairs what the robot sensed (camera images, joint angles, forces) with the actions it took, recorded at the rate the controller runs, and almost nobody records such data unless they are building a robot. Teams therefore have to produce or capture robot data themselves (1.1), which requires hardware, trained operators, physical sites, storage and quality control. The result is that data collection in robotics behaves like an operations business with research attached, and its cost is measured in people, robots and hours rather than in downloads.

Two further difficulties are less obvious.

4.2 Five sources of robot data, and three tests

Every source of robot data has a weakness, and three tests show where it lies.

The five sources are as follows.

  1. Robots practising on their own. A robot attempts tasks autonomously and records the results, often as part of reinforcement learning, in which behaviour is improved by trial and error against a reward signal. The data is complete, since every sensor and motor is logged, and it requires no human operator. Its weaknesses are cost and signal. Every sample occupies a physical robot for real time, and early random attempts at a complex task almost never succeed, so most of the data records failure and teaches little. Large installations of robot arms grasping objects around the clock produced real progress on grasping, then reached a ceiling where additional robot-hours yielded little further improvement.
  2. Teleoperation. A person controls the robot through the task, and the robot records its own sensors and commands. This gives the highest-quality data, because it is produced by the robot's own body and contains exactly the observations and actions the robot will later use. It is slow and confined to wherever a robot happens to be (4.3).
  3. Wearable capture. A person wears or holds a gripper or glove fitted with a camera, and sometimes other sensors, and performs the task directly, with no robot present. The data is far more varied and cheaper per hour, but it sits one step further from the robot: the device approximates the robot's hand, and the person's arm and body differ from the robot's (4.4).
  4. Simulation. Virtual worlds can produce trillions of examples in a day, run faster than real time on many processors in parallel, and expose quantities, such as every contact force, that are hard or impossible to measure on hardware. Their weaknesses are diversity, since building many varied and realistic virtual scenes is laborious, and fidelity, since simulated physics never matches reality exactly (4.8).
  5. Human video. Video of people doing things, from head-mounted cameras or the open internet, is extremely varied and plentiful. It is also the furthest from the robot's body. It shows a human hand rather than a gripper, it usually lacks precise 3D positions, and it carries no force information. A viewer can see a hand close on an egg but cannot see how hard it squeezes (4.5).

Because the weaknesses fall in different places, serious teams combine several sources and assign each a specific job. One common division of labour is that video teaches what a task is and how scenes and objects behave, simulation makes skills reliable by letting the robot practise under many conditions, and a comparatively small amount of real robot data supplies the precise, body-specific control. Put more simply, human data teaches the robot how the world works, and a little robot data teaches its low-level reflexes. In training terms, the broad sources tend to dominate pre-training, the initial phase in which a model learns general patterns from large datasets, while robot data dominates fine-tuning, the later phase in which the model is adapted to a specific body and task.

Ranked by quality, teleoperation usually comes first, then wearable capture, then simulation and generated data, with head-camera video last. The lower-quality sources clearly help in early training, when the model is still learning general structure. Whether they improve final performance on a specific robot, once enough robot data is available, remains genuinely disputed.

Why it matters: A robotics company's data strategy now defines it as much as its model architecture does. The informative questions about any such company are which data sources it uses, at which stage of training, and why.

4.3 Teleoperation

In teleoperation, a person controls a robot remotely and the robot records what it senses and does. Three interfaces are common. A leader-follower rig gives the operator a second, unpowered or lightly powered copy of the arm (the leader); the operator moves it by hand, and the real robot (the follower) copies its joint angles. A VR headset tracks the operator's head and hands and maps their motion onto the robot, while the robot's camera view is streamed back to the operator. An exoskeleton is worn on the operator's arm or body and measures their joint motion directly. In each case the commanded human motion must be retargeted, meaning mapped from the human's geometry onto the robot's different limb lengths, joint layout and limits, which is simplest with a leader arm that shares the robot's kinematics.

Teleoperation has a practical advantage beyond data: a teleoperated robot can deliver a useful service before it can work autonomously, and every working session produces training data. Most labs have grown their manipulation datasets this way, and the recorded demonstrations are the standard input to imitation learning, in which a model learns to reproduce the actions a human demonstrator took in each situation.

Teleoperation has four limits.

A further, technical limit is shared with all imitation learning. A policy trained only on successful demonstrations never sees what to do after a small mistake, so when it drifts slightly off the demonstrated path it enters states absent from its training data, makes larger errors, and drifts further. This compounding error, a consequence of covariate shift (the gap between the states seen in training and those met when the policy runs), is why practitioners deliberately collect demonstrations of recovering from errors as well as clean executions.

The debate: Teleoperation may be a temporary way to start or, for some tasks, a permanent business model, though in homes people are wary of a remote operator who can see and hear everything. Some teams use it as the final, high-quality layer on top of video and simulation, others mainly to verify that an arm is physically capable of a task, and few now rely on it alone.

4.4 Wearable capture

A team can dispense with the robot entirely, hand people an inexpensive capture device, and send them out into the world. One effort built thousands of these devices and collected about 500,000 hours of data, mostly of people performing everyday tasks in their own homes and workplaces across many countries (a reported result). The approach derives from an academic design called UMI (Universal Manipulation Interface), a hand-held gripper with a camera mounted on it. The camera records what the gripper sees, and the device's motion through space is reconstructed from the video, typically with visual tracking of the camera's position and orientation, so each recording yields both observations and a trajectory of gripper poses and openings that a robot can learn to reproduce.

The approach takes data collection out into the world instead of bringing the world to a robot. No robot, power supply or other equipment is needed on site, so the data covers far more homes, objects, lighting conditions and clutter than any lab could stage. Because the person holds the same kind of gripper the robot will use, the embodiment gap is narrower than with bare-hand video: the fingers and their contact with objects look the same, and only the arm and body behind the gripper differ. The lessons from that effort apply to anyone building a data operation.

Why it matters: Data collection can itself be a company's core capability, and wearable capture is a third route to robot data alongside teleoperation and simulation, trading some closeness to the robot for far greater scale and diversity.

4.5 Learning from human video

There are trillions of videos of people doing things, and robots can learn from them because people are, mechanically, biological robots operating under the same physics. How a task works barely depends on who does it. There are only one or two sensible ways to pick up a cup, and a door opens the same way for everyone. Video therefore carries information about which objects matter, the order of steps, where to grasp and how things move when handled.

Extracting that information requires bridging the embodiment gap. Common techniques include estimating the 3D pose of human hands and bodies from the video and retargeting it onto the robot's joints, predicting how objects or points in the scene will move rather than predicting the human's exact motions, and learning a compact representation of visual change that the robot's own data later grounds in motor commands. Each technique discards something: hand-pose estimates are noisy, and none of them recovers the forces involved.

Watching alone is therefore not enough. If watching were sufficient, everyone who studied a world-class tennis player would play like one. Video shows what the task is, and practice and robot data supply how to perform it with a particular body (4.2).

In reported results, one model trained on head-camera video of people, plus less than an hour of robot data, transferred the skill to humanoid robots with hands. Another model was trained on human video before its robot had even been built, on the expectation that the robot would move like a person. New tasks then required a couple of hours of robot data, and sometimes tens of minutes, such as 45 minutes for chopping with a knife.

The second robot's hardware was designed around this strategy, with body proportions measured against real people and a waist that bends, because a robot that cannot move like the people in the videos can learn less from them (5.5). The closer the robot's kinematics are to a human's, the smaller the retargeting error and the more of the video's information survives the transfer.

Why it matters: Human video is the most plentiful training data in existence, so a robot built to move like a person can draw on far more of it than one with an unusual body.

4.6 Labelling, curation and programming via data

Collecting data is only half the work. Curation is the selection, filtering and weighting of data before training, and labelling (or annotation) is the attachment of descriptive information to each example, such as the task instruction, the step being performed, whether it succeeded and how well. Three lessons make data more useful.

Why it matters: Selecting and labelling data have therefore become core engineering skills, as central to a robot's performance as the model architecture.

4.7 Data flywheel

Self-driving car programmes demonstrated the approach first. A team builds a system useful enough to deploy, puts it to work, and lets it collect data as it operates, instead of trying to calculate in advance how much data it will need. The difficulty is circular: data is needed to build a robot that works on its own, and deployed robots are needed to gather that data. A data flywheel breaks the circle by letting each round of deployment produce the data for a better next version, which can then be deployed more widely and produce more data. Deployment is valuable not only for volume but because it surfaces the rare situations, often called the long tail, that no planned collection anticipates.

The robotics flywheel is harder to start than the self-driving one, for four reasons.

Self-driving took 17 years to reach a few thousand driverless cars, partly because cars are so specialised: the system must be nearly perfect at one high-risk task before it can be deployed at all. A child allowed to learn only driving would learn slowly. A general-purpose robot can begin with low-risk tasks, where occasional errors are tolerable, and build up to higher-risk ones, much as people do.

Training thus gains a third stage after pre-training and fine-tuning: learning from deployment data, which keeps accumulating and over time will outweigh everything else.

Why it matters: Each pilot should be planned as the start of a flywheel: it should prove the return on investment, produce the data the next capability needs, and let the customer see its own robot improve. The order of pilots matters, because each one supplies data for the next.

4.8 Simulation and sim-to-real

Simulation means training or testing a robot in a virtual world governed by a physics engine, software that computes how bodies move, collide and exert forces on one another at each small time step. It is used because real-world data is slow, expensive and sometimes dangerous to collect, while a simulator can run many robots in parallel, faster than real time, and reset instantly after a fall. The sim-to-real gap, also called the reality gap, is the difference between the simulator and reality, which means a skill learned in simulation may fail on the real robot. Its main sources are well known: contact and friction are hard to model accurately, motors and gears have delays, backlash and saturation that simulators simplify, rendered images differ from real camera images, and soft or deformable objects such as cloth and cables are expensive to simulate at all. A perfect simulator would amount to having roughly solved the universe, so the gap never fully closes.

Practitioners narrow the gap in two complementary ways. System identification measures the real robot's physical parameters, such as masses, friction coefficients and motor response, and tunes the simulator to match them. Domain randomisation does the opposite: it deliberately varies those parameters, along with lighting, textures, object shapes and sensor noise, across a wide range during training, so the policy learns to succeed under all of them and treats the real world as one more variation. Domain randomisation is the main reason simulated locomotion policies transfer to hardware without real-world training data.

Teams use simulation in three main ways.

The big open debate: Humanoid walking and acrobatics rely heavily on simulation, often with no real-world data at all, while manipulation tasks such as folding and making coffee mostly use real data with large models, though some teams now bring simulation into manipulation too. For a company with limited capital, starting in simulation reduces risk before hardware is built or modified.

4.9 Simulation spectrum and world models

"World model" has at least 20 definitions in current use. It helps to treat them as points on a spectrum of methods for creating a virtual world, ordered from hand-built and explicit to learned and implicit.

  1. Classical physics simulation. People build 3D worlds by hand, with modelled geometry, materials and physics, much as film studios build computer-generated imagery. This is precise and controllable, and every quantity is available for measurement, but creating varied, realistic scenes is expensive in human effort.
  2. Gaussian splatting. Camera footage of a real place is converted into a 3D scene represented as a large collection of small, semi-transparent, coloured blobs (3D Gaussians) whose positions, shapes and colours are optimised until rendered views match the footage. A virtual camera can then move through the scene and render photorealistic views from new angles. The scene comes from reality but remains an explicit 3D model, and it captures appearance rather than physics, so it is typically combined with a physics engine for interaction.
  3. Neural or generative world models. A neural network directly generates video or sensor frames, with no explicit 3D scene, by predicting what comes next from what came before. Some are action-conditioned, meaning they take the robot's intended action as an input and generate the consequences of that action. Their realism comes from training on large amounts of video, and their characteristic failure is physical implausibility: objects that drift, merge or vanish over long rollouts.

A working definition: a world model is a simulation close enough to reality, and responsive enough, that a robot can act in it and the world reacts appropriately. A reactive (closed-loop) world responds to what the robot does, so a policy can be tested or trained inside it. An open-loop world simply replays a recording regardless of the robot's actions, which is useful for evaluating perception but cannot show the consequences of a different decision.

The current role: World models used to run only offline, for development and testing, and some now run inside the robot's live decisions. One recent vision-language-action model (VLA) uses a world model while it works, to picture where the robot should be a second or two ahead, and then chooses actions that move it toward that predicted state. World action models predict future video as part of choosing their actions (3.5). A robot has also been shown imagining training scenarios inside its own model, described as the robot dreaming, to extract more learning from less than an hour of real data; the underlying idea, training a policy on experience generated by a learned model rather than by the real world, is long established in model-based reinforcement learning. Two cautions apply. The imagined images help perhaps less than one might expect, because errors in the model become errors in the training data, and enthusiasm for world models can distract from the underlying constraint, which is still data.

Why it matters: The phrase "world model" should always prompt two questions: which kind is meant, and whether it runs offline to create training data and test the robot, online inside the robot's decisions, or, increasingly, both.

4.10 Evaluation

Robotics has nothing like the fixed, shared test sets of vision and language, where a model can be scored on the same examples by anyone, quickly and cheaply. A robot policy must be evaluated by running the physical robot many times, because its own actions change what it sees next, so a recorded dataset cannot stand in for a trial. Each trial takes real time and often a human to reset the scene, and results vary with lighting, object placement and wear, so many trials are needed before a difference between two models is statistically meaningful. Failures also compound over long tasks: a task made of many steps succeeds only if every step does, so evaluating a 20-minute task is more than 10x harder than evaluating a 2-minute one. The number of robots available caps how fast a team can test, and a team with few robots learns slowly. In academic robotics, teams mostly grade their own homework, choosing their own tasks, setups and success criteria, which makes results hard to compare across papers.

Simulated benchmarks offer repeatability and scale, but policies that perform well in a simulator do not reliably perform well on hardware, for the reasons given in 3.8, so they serve better for rapid iteration than for final judgement. Teams with commercial goals now use tougher tests.

Why it matters: Short tasks are cheaper to test and easier to get working, so they are the right place to start. Testing infrastructure deserves early investment, and progress should be measured against the customer's standard.


4.11 What the benchmarks show

Most robot results are measured on a team's own tasks, so two results are rarely comparable. A shared benchmark runs every model on the same tasks under the same scoring. One of the most complete in 2026 is RoboDojo: 42 manipulation tasks in simulation and 18 on real robots. Instead of one success rate, it scores five separate abilities:

Each ability gets a score out of 100 (partial credit for progress) and a success rate (full completions only). The overall score is the plain average of the five.

By October 2026 the board held 62 entries, and four things stood out:

  1. Manipulation is far from solved. The best purpose-built robot model averaged about 39 out of 100 and fully completed about a third of attempts.
  2. Progress is fast. Open robot models from 2024 and 2025 score between 0.2 and 3.5. On a related simulation suite, the best 2026 models average 84% success, against 14 to 31% for 2025 baselines.
  3. Open instructions are the weak spot. Most robot models score close to zero on the Open ability. They do what they were trained on and little else.
  4. The top of the board is hybrid. The highest scores, around 55, come from agent systems: a language model plans and checks the work, and learned robot skills carry out the motion.

A September 2026 evaluation tested what happens with no robot model at all. A frontier general-purpose language model, with no robot training, controlled the arms through a simple fixed interface with two commands: move the gripper to a point, or give up. In simulation it scored 29 overall with 22% success, above every purpose-built robot model on the board at that time (the best was 24.9). The previous generation of the same model scored about 1 under the same setup.

Behind that average, the model reached 50% success or better on ten tasks and zero on sixteen. It won where the task is mostly reasoning: reading numbers, following a rule given in words, or working out a geometric arrangement. It lost where the task is mostly contact: inserting, stacking under friction, pouring.

The study concluded that the model now understands what to do and where things are in 3D, but lacks physical commonsense: a working feel for contact, force, friction and how things move once touched. On real robots the gap was obvious. Testing was stopped for safety after the model repeatedly issued unreasonable or unsafe motions, some of which damaged hardware. Of 33 real trials kept for analysis, one fully succeeded.

Two further results from the same study matter for how robots will learn:

An October 2026 study gave nine frontier language models (fifteen configurations) direct control of a real two-arm robot on six tasks, with 20 trials per task and a cap of 40 moves per trial. The best models averaged 45 to 48% on a partial-credit score, close to a tie. The scores reflect steady partial progress on each task rather than frequent end-to-end success. They have also been rising quickly:

No single model was best on success, speed and cost at once. A common failure was closing the gripper on nothing: weaker models did it 59 to 64% of the time, stronger ones 32 to 39%. Some tasks stayed out of reach: standing bottles upright succeeded in 6 of 180 trials. The study lists its own limits: one robot, six tasks, a move cap, grading done by a language model watching the video, and trends fitted on few data points.

Why it matters: General-purpose models are getting better at robotics quickly, without any robot data, on the reasoning half of the problem. The contact half, which covers precision, force and dynamics, still depends on robot data and purpose-built models, and that is where most paid physical work sits (11.2). Public boards show the direction of progress, but deployment decisions still rest on the customer's own tasks (4.10).

← Part 3. Robot intelligence   Part 5. Hardware →