Physical AI, Explained · Part 3
Robot intelligence
Read this part in the book →A robot model is a learned function that maps camera images, joint positions and an instruction to motor commands. It acquires that mapping from human demonstrations and from its own practice, and the current evidence is that models trained across many robots and many tasks outperform models built for a single one. This part explains what such a model consumes and produces, how the main families differ, how they generate motion, how they learn, and where their limits currently lie.
3.1 Capable bodies with no brain
For decades the mechanical side of robotics was, for most purposes, adequate. Industrial arms could already move with sub-millimetre repeatability, carry heavy payloads and run for years. What they could not do was cope with a world that had not been arranged for them. A classical industrial robot replays a programmed trajectory; if the part arrives rotated or a box is left in its path, it has no way of noticing, let alone adapting. The missing ingredient was the intelligence to perceive a scene, interpret it, decide what to do and act on that decision while the scene changes.
Earlier attempts to supply that intelligence relied on hand-written rules and explicit geometric models of objects and environments. These worked inside controlled cells and broke down in open settings, because no engineer can enumerate every situation a robot will meet. The modern field rests on a different bet: that this intelligence can be learned from data, in the same way that language and image models learned their abilities, rather than programmed case by case.
3.2 Three sub-problems: semantics, planning, control
A useful robot needs three abilities working together. Each is partly solved in isolation, and most of the remaining difficulty lies in combining them so that a failure in one does not undo the others: - Semantics means understanding the instruction and what is in the scene: that "the mug" refers to the blue object on the left, that a towel is soft and a glass is fragile. Large language models and vision-language models made rapid progress here, because this kind of knowledge is abundant on the internet. - Planning means putting steps in the right order to reach a goal. Clearing a table requires picking up the cups before wiping the surface, and opening a drawer before placing something in it. Planning errors are often invisible in short demos, which rarely require more than a few steps. - Control means moving in real time while the world keeps changing. The robot repeatedly measures what actually happened, compares it with what it intended, and corrects the difference. This feedback cycle is called closed-loop control, in contrast to open-loop control, which executes a planned motion without checking the result. Closed-loop control is what allows a robot to keep its grip when an object turns out to be heavier than expected, or to adjust its foot when a step is slightly higher than predicted.
The third ability is often called sensorimotor common sense: the continuous coupling of perception and movement that lets a body adjust to its surroundings without deliberate thought. Humans tend to undervalue it because nearly everyone has it, and because it operates below conscious awareness. It is also the ability for which the internet provides almost no training data, since text and video record what people did, not the forces they felt or the corrections they made.
The three abilities operate on different timescales. Semantic understanding may be updated once per instruction, planning every few seconds as sub-tasks complete, and control many times per second. Many system designs mirror this split, with a slower reasoning component setting goals for a faster component that produces motion.
Why it matters: A claim that "the robot works" should prompt the question of which of the three abilities is meant, since a polished demo can conceal a serious weakness in any one of them.
3.3 The case for general models
The leading research groups are betting on robot foundation models. A foundation model is a single large model, trained on broad data, that can be adapted to many downstream uses; in robotics, the aim is one model that controls many kinds of robots and performs many tasks, instead of a separate system engineered for each. The idea comes from language processing. Researchers first built narrow systems, one for translation, another for sentiment analysis, another for question answering, each trained on its own curated dataset. General language models trained on vast amounts of text then outperformed all of them. Training on everything produced a broad internal representation of language and the world on which specific skills could build.
For a sufficiently difficult intelligence problem, enlarging the problem can make it easier. A model trained on all of language translated better than a model trained only on parallel translations, because it had learned grammar, idiom and world knowledge that the translation data alone could not supply. The effect is expected to be stronger in robotics, because physical situations vary so widely that any single task's data covers only a small fraction of what a robot might encounter, while a general model sees far more unusual situations across its training set.
Three further arguments favour generality: - Single-task robots stall. A robot built for one job, such as welding, typically reaches 80 to 90 percent performance quickly. It then meets rare situations, such as a box left in the way or the lights going off. A single task never produces enough examples of these events to learn from, so a person has to remain nearby to handle them, and the economics stop working. The last stretch of reliability is where most of the cost of automation sits, because each remaining failure mode is individually rare and therefore expensive to collect data for. - Rare cases transfer between settings. The corner case of one industry is often the central case of another. An object falling from a shelf may be rare in a factory and routine in a warehouse; a reflective or transparent item may be unusual in one kitchen and common in a laboratory. A model trained across many settings can learn each setting's rare cases from another setting's everyday data. - Recovery skills transfer. In one experiment, a robot that recovered from a mess it had never seen drew much of that ability from recovery data recorded on other tasks. Recovery behaviours, such as regrasping a slipping object or backing away from a collision, share structure across tasks even when the tasks themselves differ. A model trained on one task sees few such episodes and therefore copes worse with surprises.
The practical pattern that has emerged resembles the language field: a general model is pretrained on large, varied data and then fine-tuned, meaning trained further on a smaller set of examples from the target task, so that it specialises without discarding what it learned during pretraining.
Why it matters: Companies increasingly build or adopt one general model and adapt it to each task, from dishes to laundry, rather than engineering a new system per application.
The debate: General models can outperform specialists without a single general robot winning every market, and both general and task-specific robots are likely to be large markets, much as specialised language models for code or mathematics are regaining ground. Whether the general model itself becomes a company's durable advantage is a separate question (7.9).
3.4 What a robot model takes in and puts out
- Inputs: camera images (sometimes supplemented by depth or lidar data, which is usually converted into an image-like grid as well), the robot's own joint angles, and an instruction, usually in natural language. The robot's sense of the position of its own joints is called proprioception, by analogy with the human sense that allows a person to touch their nose with closed eyes. Some systems also take in force or touch readings, the gripper's opening width, or a short history of recent actions.
- Outputs: a vector of numbers specifying where each joint or gripper should move next. A typical model outputs roughly 30 to 50 numbers per step, and one semi-humanoid robot uses about 100. That robot's model produces a new set of numbers 10 to 20 times a second. One walking model processes vision and produces actions 100 times a second.
Before a transformer-based model (3.6) can process these inputs, each must be converted into tokens, the fixed-length vectors that the model operates on. Text is split into word fragments, each mapped to a learned vector. Images are typically cut into a grid of small square patches, and each patch is projected into a token, so a single camera frame becomes a few hundred tokens. Joint angles are scaled into a standard range and projected into tokens of their own. The model then attends across all of these tokens together, which is how it relates the word "cup" to a region of the image and to the current position of the arm.
The output side involves a design choice called the action space: what the numbers represent. The common options are target joint positions; a target pose for the gripper in space (its position and orientation), which a separate inverse kinematics routine converts into the joint angles that achieve it; joint velocities; or, more rarely, joint torques. Most learned models output target joint positions instead of driving the motors directly. A small classical controller then pushes each motor toward its target hundreds of times a second. This controller is usually a PD controller (proportional-derivative): it applies a torque proportional to the gap between the target and the current position, minus a term proportional to the joint's velocity that damps oscillation. The rule is older than modern robotics and needs no learning.
This layering resolves a timing mismatch. A large model cannot run hundreds of times a second, but the motors need commands at that rate. The model therefore sets targets at its own slower rate, and the low-level controller interpolates and tracks them in between. The choice of output level does not currently appear to be a bottleneck; systems using joint targets and systems using gripper poses both reach high performance.
Some teams build strictly end-to-end systems, which take camera pixels in and send motor targets out, with no separate modules for object detection, mapping or motion planning in between. Any planning happens implicitly inside the model, learned from the data. The advantage is that no hand-designed interface between modules can throw away information the task needs; the cost is that failures are harder to diagnose, because there is no intermediate output to inspect.
Why it matters: Every robot "brain" in a demo is this mapping from inputs to numbers, and what separates good models from poor ones is whether they produce the right numbers in situations they have never seen.
3.5 Three families of robot model
The leading teams build robot models in three distinct ways and disagree about which will prevail.
1. Vision-language-action models (VLAs). A VLA is a large language model adapted to control a robot. It is built in three stages: 1. Train it on text, so it learns language and general knowledge. 2. Train it further on web images paired with text, so it learns to connect words to what it sees. A model at this stage is called a vision-language model. 3. Train it further on varied robot data, so it learns to output actions instead of words.
VLA builders start from a language model because the two kinds of knowledge complement each other. Robot data comes from the real world but contains little general knowledge: a few hours of arm motions say nothing about what a colander is for. A language model contains a great deal of knowledge but has never touched anything. Each supplies what the other lacks. In practice the third stage must be handled carefully, because training heavily on robot data can erode the knowledge acquired in the first two, an effect known as catastrophic forgetting. Builders mitigate it by continuing to mix web data into robot training, or by attaching a separate action-generating component so that the language backbone is disturbed less.
There are two main ways to make a language model emit actions. The first discretises each action dimension into a fixed set of bins and treats each bin as a token, so the model "writes" an action exactly as it would write a word. The second attaches a dedicated action expert, a smaller network that reads the language model's internal representation and produces continuous actions, usually with diffusion or flow matching (3.6). The second approach generally yields smoother and more precise motion at high control rates.
VLAs have improved in clear stages. The first versions joined a vision-language model to an action generator (see 3.6) and were never meant to be final. The next versions could split a large instruction such as "clean the kitchen" into steps before acting, and worked in homes they had never seen. After that, teams fine-tuned versions for specific jobs like making espresso or building boxes, then improved them with reinforcement learning. Most recently, a single general model matched or beat those fine-tuned specialists with no extra training (a reported result).
2. World action models. This approach holds that a robot should predict what will happen next in the world, not only its own next move. A world model is a model that predicts how an environment will evolve, often conditioned on an action. One such model predicts future video of the scene together with the robot's actions, and was trained on more than a million hours of video, mostly of people (a reported result). It works like a language model reasoning step by step, except in pictures: the model imagines how the scene will change, then acts to bring that future about. The reasoning is that any physical system has to decide what to do next, which requires predicting the future; the robot is part of the world, so the model should predict as much of the world as it can. A practical attraction is that video of people is plentiful while robot data is scarce, so a model that learns physics from human video can draw on a much larger pool. The open difficulty is that generating video is computationally expensive, and the model must still translate imagined scenes into the specific commands of a particular robot body.
3. Models trained mainly on physical data. This approach trains the model almost entirely on physical data. In one reported case, about 99 percent of the model was trained from scratch, keeping only about 1 percent from a pretrained model. On this view, language pretraining is a crutch once enough physical data exists, much as reading about skiing never makes anyone a good skier. Physical interaction is the foundation, and language is added on top so that people can give the robot instructions.
The debate: VLA advocates argue that web knowledge lets a robot handle the unexpected; world-model advocates argue that "actions" are an abstraction carrying human assumptions and that the bitter lesson, the observation that simple methods with more data beat hand-designed ones, favours predicting raw video; physical-data advocates argue that at enough scale such data teaches dexterity better than web text. A fourth position cuts across all three: the data matters more than the architecture, so the data available now, and the data the robot will generate as it improves, should determine the model design.
3.6 How the models make smooth, precise motion
Two technical terms appear in almost every description of a current robot model.
- Transformer is the neural network design used in large language models. Its core operation, attention, lets every token in the input weigh its relevance to every other token, so the model can relate a word in the instruction to a region of an image and to the arm's current position in a single computation. Transformers keep improving as they are given more data and more parameters, which is the main reason they have spread across fields as different as gene therapy and robotics, applied to very different data.
- Diffusion, and its simpler relative flow matching, are methods for generating an output by starting from random noise and refining it over a series of steps until it is sharp and precise. Image generators use them to produce crisp edges. Robots use them because their actions also need to be precise and smooth.
The reason a robot needs a generative method, rather than a network that simply outputs one best guess, is that demonstrations are often multimodal: there are several valid ways to perform the same action. Asked to move around an obstacle, one demonstrator goes left and another goes right. A network trained to minimise average error on such data learns to predict the average of the two, which drives the arm straight into the obstacle. A generative method instead learns the full distribution of valid actions and samples one coherent option from it.
Diffusion achieves this as follows. During training, the model is shown real action sequences from demonstrations, corrupted with varying amounts of random noise, and learns to predict the noise that was added (equivalently, to recover the clean actions). At run time it starts from pure noise and repeatedly removes the predicted noise, conditioned on the current camera images, joint positions and instruction, until a clean action sequence remains. Flow matching reaches the same goal more directly. It learns a velocity field that transports noise to data along paths that are close to straight lines, and generation consists of following that field from a noise sample to an action. Straighter paths can be followed accurately in fewer steps.
Most of these models also use action chunking: instead of predicting one action at a time, they predict a short sequence of future actions in one pass, execute part or all of it, and then predict again from the new observation. Chunking has three benefits. It produces smoother motion, because consecutive actions are generated together and are therefore consistent. It reduces the number of separate decisions at which an error can creep in. And it copes better with demonstrations in which the person paused or hesitated, which would otherwise confuse a model predicting one step at a time. Some systems overlap successive chunks and blend them to avoid a visible jolt at each boundary.
Combining the two ideas gives a diffusion transformer: the transformer interprets the inputs and learns from large amounts of data, and the diffusion or flow-matching process turns its output into fine, smooth motion. Most of the field now uses some version of this combination. Flow matching has become especially popular because it needs less tuning and fewer refinement steps, which matters when a robot must recompute its actions many times a second on hardware that also has to run everything else.
3.7 Imitation vs. reinforcement learning
- Imitation learning, also called behavioural cloning, means recording demonstrations of a task, usually a person controlling the robot remotely (teleoperation), and training the model to reproduce the demonstrated actions from the same observations. Almost every system starts this way, because it is the most direct way to give a model a working behaviour. It tends to top out near human level, because the model is copying people, mistakes included. In one case, robots learned to always miss first and then pick up the object, because the people giving demonstrations sometimes fumbled and tried again.
- Reinforcement learning (RL) means the robot attempts the task itself, receives a numerical signal called a reward when it does well, and adjusts its behaviour to increase the total reward it collects. RL can exceed human performance because it is not limited to moves a person made. The famous example is "move 37," a move by a Go-playing program that no human player would have chosen.
Imitation learning has a characteristic failure mode called compounding error. A cloned policy is never perfect, so at each step it deviates slightly from what the demonstrator would have done. Those small deviations carry the robot into states that never appeared in the demonstrations, because the demonstrator never made those particular mistakes. In such unfamiliar states the model's predictions are worse, so it deviates further, and the errors accumulate over the course of a task. The underlying mismatch between the states seen in training and the states encountered at run time is called covariate shift. The standard remedy is to collect data in the states the robot actually reaches: a person watches the robot run, takes over when it drifts, and the corrective actions are added to the training set. This interactive procedure, known as dataset aggregation, teaches the model how to get back on track from its own typical mistakes. The "miss first" example illustrates a related problem: imitation copies whatever is in the data, so practitioners filter poor demonstrations or label each one by quality so that the model can be asked to imitate only the good ones (3.8).
RL has its own difficulties. The first is reward design. A sparse reward, given only when the task succeeds, is easy to specify but gives the robot almost no information during the many early attempts that fail. A dense reward, which scores partial progress such as distance from the gripper to the object, speeds learning but can be exploited: the robot may find a way to increase the score without doing the task, a behaviour called reward hacking. A common modern approach is to train a separate model to judge from camera images whether a task has succeeded or how much progress has been made, and to use its judgement as the reward. The second difficulty is sample efficiency: RL typically needs very many attempts, which is cheap in simulation and slow, expensive and potentially damaging on real hardware, where a person may also have to reset the scene after each attempt. The third is safe exploration, since trying random movements near people or fragile objects is unacceptable. For these reasons much RL is done in simulation, with domain randomisation (varying friction, mass, lighting, sensor noise and other simulator parameters during training so that the real world looks like just another variation) used to make the learned behaviour survive the transfer to a physical robot, the sim-to-real gap.
Over a robot's working life the two methods combine. First people teach it, then people correct it, and once it is clumsy but autonomous, RL removes the clumsiness. The division of labour is precise: a human correction teaches the robot how to recover after dropping something, while practice teaches it to stop dropping things. Teams are now finding ways to make real-world RL affordable (Part 6).
The grand challenge: The two largest results in AI are generative AI and deep RL. Generative AI, such as large language models, reproduces human abilities and holds an enormous amount of world knowledge; deep RL, such as the Go-playing programs, finds new solutions that surpass humans. The current frontier in robotics is combining them: begin with the world knowledge of generative AI, then use RL to go beyond human performance.
Why it matters: The mix a team bets on determines where its money goes: progress driven mostly by demonstrations calls for investment in data collection, while progress driven by a few demonstrations plus extensive practice calls for investment in deploying robots and letting them practise safely.
3.8 Reasoning before acting
The hardest part of the real world is the long tail of unusual situations that no training set anticipated. People handle them with common sense learned elsewhere. Robots now obtain some of this from language models, and several techniques have emerged for putting it to work.
- Step-by-step reasoning. Before acting, the robot generates a written plan for itself. Told to clean a kitchen, a reasoning model first produces "pick up the dish, pick up the towel, wipe the spill," and then executes each step. Coding assistants do the same, writing a plan before writing code. This technique is called chain-of-thought. In robotics it is often implemented hierarchically: a high-level component, which may run only every few seconds, produces the next sub-task in words, and a low-level component converts that sub-task into motion at the control rate. Writing the sub-task down has a further benefit, in that a person can read it and see why the robot did what it did.
- Teaching with words. Because the model works in named steps, a new skill can be taught by describing its steps, without recording any new motion data, provided the robot already has the underlying motions. A robot has been taught to use an air fryer this way.
- Reasoning with pictures. Some information is hard to convey in words. "The power button is about three inches left of the edge" helps little, while a generated picture of the robot's finger on the button specifies exactly what to do. Recent models can generate images of where the robot should be a second or two from now, currently using a separate image model, and then act to reach that pictured state. The predicted image serves as a precise visual sub-goal.
- Detailed labels. In one model, labelling every training example in detail, with the current sub-step, how good the demonstration was and how fast it was performed, made a large difference. Video generators work the same way when they expand a short prompt into a long description: a more detailed prompt carries more information about the desired output, so the model learns more precisely what the right answer looks like. Quality labels also address the imitation problem described in 2.7, since at run time the model can be conditioned on "high quality" and so imitate the best demonstrations rather than the average one. In that case, detailed labelling mattered more than making the model bigger.
The main cost of these techniques is latency. Generating text or images before acting takes time, so most systems reason at the slower, higher level and leave the fast control loop untouched.
Why it matters: No team can collect data for every situation, so words and pictures become inexpensive ways to steer a robot alongside demonstrations.
3.9 Compositional generalization
A capable language model can write whole paragraphs in the phonetic alphabet, even though dictionaries show that alphabet only one word at a time. Combining known pieces in a new way like this is called compositional generalization. An image generator can likewise draw an armchair shaped like an avocado. The model never needed a picture of that exact combination, because it had learned "armchair" and "avocado" as separable concepts that can be recombined.
The distinction that matters here is between interpolation, producing something similar to what was seen in training, and composition, assembling familiar parts into a configuration that never occurred. Classical robot learning rarely achieved the second: a policy trained on one object in one place often failed when either changed. Robots trained on broad data now show it: - A rare appliance. One general model used an air fryer that appeared in only three training examples, by combining skills it already had: open, place, close. - A known task on a new robot. A large industrial two-armed robot folded a shirt with no folding data from that robot, and came close to the quality of a person controlling it remotely (a reported result). The folding skill came from other robots and the control of this body from other tasks. - Switching hands. A robot trained to always hold a pin in its right hand ended up with the pin on the wrong side, and used its left hand instead. Nothing like that was in its training data. - The same effect elsewhere. On other robots, one taught to shake a bag with its left hand once used its right. Another robot, knocked with a hockey stick, reached across with its other hand to catch a washer in mid-air.
Composition depends on the model having learned representations in which skills, objects and bodies are somewhat independent of one another. It is not guaranteed, and current models still fail at combinations that seem trivial to a person, so these results show that the capability exists, not that it is reliable.
Why it matters: Composition is the reason general models pay off, because data collected for one task can improve tasks it was never collected for.
3.10 Cross-embodiment transfer
Embodiment means a specific robot body, with its own sensors, motors, joints and proportions. Cross-embodiment means one model controlling many different bodies. Surprisingly, a single model has worked on robots with different numbers of joints, and even on multi-fingered hands, without being told which robot it was running on.
The technical obstacle is that bodies differ in their degrees of freedom (the number of independent ways a robot can move, roughly one per joint), their camera placements, their joint limits and their dynamics. Practitioners handle this in a few standard ways. Action vectors from different robots can be padded to a common length, with unused entries ignored. Actions and joint readings can be normalised to a shared range so that one robot's large numbers do not dominate another's small ones. Some designs give each embodiment its own small input and output layers around a shared core, so that the core learns skills common to all bodies while the edges handle each body's specifics.
Neuroscience suggests why a body-agnostic model is plausible. When monkeys use tools, neurons that track the position of the hand begin to respond at the tip of the tool. The brain treats the tool as part of the body, which implies that a good robot brain should not depend heavily on which body it inhabits. The strongest evidence comes from the Open X-Embodiment project, which pooled data from many institutions. One general model trained on about 10 types of robot beat 10 specialist models, each tuned to one robot, by 50 percent on the specialists' own robots. Part of the explanation is that even a single robot changes over time, through worn motors, recalibrated cameras and hardware revisions. Training on many bodies teaches the more general skill of controlling whatever robot the model finds itself in.
Some teams take this as far as it goes, with one model meant for any robot and any task. Their argument is partly about data, since robot data is so scarce that no robot's data can be wasted, and partly about safety. When a joint breaks, a 32 degree-of-freedom robot becomes a 30 degree-of-freedom robot. A model tuned to one exact body fails at that point, while a model trained on many bodies treats the damaged robot as a new one and adapts. In reported demos, a four-legged robot with a failed knee keeps walking on three legs within milliseconds, a dog-like robot with useless front legs stands up and walks on two, and a wheeled robot with jammed wheels starts walking.
The model works out its new body from the last few moments of experience, without retraining. Given a short history of the commands it sent and the motion that resulted, it infers how this body responds and adjusts accordingly, an implicit form of what control engineers call system identification. This is called in-context learning, the physical counterpart of a language model picking up a pattern from examples in its prompt: the model's weights do not change, only the context it conditions on. The ability was reportedly not designed in; it was noticed when a robot kept walking with broken ankle motors.
Why it matters: Data and models should be planned for many robot types from the start, since a model tied to one body has no fallback when a joint fails.
The debate: Some teams deliberately master a single robot body before attempting cross-embodiment, which is a decision about focus rather than evidence against the approach. The humanoid race shows the cost of skipping it: humanoid hardware is meant to be general, yet each company's model works only on its own humanoid.
3.11 Memory and context
Most state-of-the-art robot foundation models have no memory or context. They act only on what the cameras show at that moment, a property called being memoryless or reactive. That suffices for short, repetitive motions, where the current image contains everything needed to choose the next action, but long jobs require memory. Halfway through opening a microwave, a single frame cannot show whether the robot is opening or closing it. A dryer's blinking indicator may appear off in one frame even though the machine is running. In both cases the correct action depends on information that is no longer visible, and a memoryless policy can oscillate, repeat a completed step or abandon one halfway.
Remembering everything is too expensive. Ten seconds of video, at 50 frames a second, from four cameras, at about 256 tokens per image, comes to around half a million tokens. That is far too much to process in real time, particularly because the cost of attention in a standard transformer grows with the square of the number of tokens: doubling the context roughly quadruples that part of the computation. Even at one frame a second the total is about 10,000 tokens, and that still covers only 10 seconds. The constraint is latency as much as cost, since a robot that takes too long to decide is acting on a scene that has already changed.
One solution keeps memory at two levels of detail, much as a person remembers the last few seconds vividly and the last quarter of an hour as a sequence of events. The last ten seconds are stored as compressed video, preserving the visual detail needed for fine motion. Anything older becomes a short text summary covering the past 10 to 15 minutes, which records what has been done ("dishes rinsed, counter wiped") at a tiny fraction of the token cost. With this setup, robots finished 10 to 15 minute kitchen clean-ups on their own, including wiping, drying, putting things away and washing dishes (a reported result).
Why it matters: Together with reliability, memory is what separates a 30-second demo from a robot that can do a real job.
3.12 Data and scaling laws
Scaling laws are empirical relationships showing that a model's performance improves predictably, typically as a smooth power-law curve, as training data, computing power and model size increase. In language models they allowed researchers to forecast the benefit of a larger training run before paying for it. The Open X-Embodiment project described in 2.10 found scaling laws for robotics, and they are now starting to appear more clearly, but slowly. Because the curves are smooth only when plotted on logarithmic axes, data must grow by orders of magnitude before a scaling law becomes visible, and each 100x increase in robot data takes about a year to collect. Progress is nonetheless real at the level of individual deployments: in one reported case, the robot data needed for a live factory task fell from more than a month to under 24 hours within a year.
Data is the limit, for four reasons: - The gap is very large. One estimate puts the data a robot foundation model needs at around 100 trillion examples. The largest robot teleoperation datasets hold around a million, so almost the whole 100 trillion is missing. The figures are one company's estimate, and the conclusion holds even if they are far off. - More data does not mean more variety. In language, more data brings more variety automatically, because the internet covers so much of human life. In robotics it does not. Trillions of hours of self-driving data are still all driving. A hundred robots in one room learn only about that room, and 100 trillion tokens are useless if they all come from the same environment. - Variety matters more than volume, and this has been measured. When the most varied data was removed from one training set, performance on new tasks dropped sharply. Removing a random fifth of the less varied data barely changed anything. - Even unrelated data helps. Adding data unrelated to a task, even data collected in another country, has been reported to improve performance on it, with no sign of the effect levelling off.
Data can be scarce in two distinct ways. In some cases the data does not exist and has to be generated, through teleoperation, simulation or the robot's own practice. In others it is being produced somewhere, for example in the motions of workers or in the logs of existing machines, but nobody has built a way to capture it in a form a model can learn from. Each source has a characteristic weakness. Teleoperation is accurate but slow and labour-intensive. Simulation is cheap and unlimited but suffers from the sim-to-real gap (3.7), which is widest for contact-rich manipulation. Video of people is abundant but contains no motor commands or forces and shows human hands, not robot grippers. Most serious efforts therefore combine several sources.
Why it matters: Data, more than algorithms or hardware, is the main bottleneck, with one caution: even if all the data appeared tomorrow, today's algorithms might not be able to use it.
3.13 Moravec's paradox
Tasks that are easy for people, such as picking up a cup or climbing stairs, are hard for machines, while tasks that are hard for people, such as calculus or chess, are easy for machines. This inversion is known as the paradox in this section's title. A computer beat the world chess champion in the 1990s, yet no robot can reliably pick up the pieces and set up any chess board.
There are two standard explanations: - Evolution. People are excellent at physical tasks because ancestors who were poor at them did not survive. Abilities refined over that long a period are deeply optimised and run without conscious effort, so tasks that feel effortless in fact rely on a vast amount of hidden processing. Abstract reasoning is recent in evolutionary terms, which is why it feels hard to people and why its explicit, rule-based structure suits computers. - Interaction with an uncertain world. A backflip is complete control of one's own body in free space. Computers excel wherever everything is known and controlled. Airplanes have flown on autopilot for decades because in the air they touch nothing, and their dynamics can be modelled accurately. The hard part is continuously sensing an uncertain world and correcting for it, such as placing a foot on a stair never seen before or closing a hand around an object of unknown weight and friction.
Contact is the core of the second explanation. When a hand touches an object, the forces change abruptly, friction is hard to predict, and the hand itself hides the object from the cameras at exactly the moment the details matter. These effects are also difficult to simulate accurately, which is why simulation transfers less well to manipulation than to motion in free space.
Two practical observations add to this: - Walking corrects itself, but each grasp is different. Walking works like a pogo stick. The same motion repeats, recovers on every step and absorbs variations in the ground. Each grasp is a separate problem, beginning with considerable uncertainty about the object's shape, position and weight and ending with the object held, and there are many valid ways to achieve it. - Legs get more practice signal than hands. A walking robot that tries random movements falls in every possible way, and every fall teaches it something, since the outcome is clear and the dynamics are continuous. A hand touches an object only briefly, so most random attempts produce no contact at all and teach it little. That is why simulation plus RL solved walking before it solved grasping.
The same logic applies to evaluating demos. Dancing, martial arts and backflips replay a planned motion in free space, so they are easier than they look, and so is blind parkour, which relies on proprioception rather than vision. Stairs are the honest test, especially descending stairs while being pushed, or steep fire-escape stairs, because the robot must see and react. A humanoid that cannot climb stairs has little reason to have legs.
How much error a task tolerates is the other test. Folding laundry is relatively easy for learned robots because a shirt still looks fine if a fold is off by a centimetre, and cloth forgives imprecise contact. Putting a wireless earbud into its case with a two-fingered gripper is hard, because there is no room for error and the gripper cannot reorient the part in the final moment, as human fingers do without thought.
Why it matters: The two tests, the need to react and the tolerance for error, are better guides than intuition when choosing which tasks to automate first, since how hard a task looks to a person predicts little.
3.14 Where the bottleneck is moving
Some models improved simply from adding high-level language labels to existing data, with no new demonstrations. The main limit had moved up from the lowest level, whether the robot can physically perform a motion, to the middle level of interpreting the scene and choosing the right next step, a level that language can supervise. A later model confirmed the trend, since more detailed descriptions of the data helped more than a bigger model did (3.8).
The shift changes which failures dominate. Robots in current systems more often fail by choosing the wrong sub-task, forgetting what they have already done or losing track of a long sequence than by being unable to execute a grasp they have chosen. These are problems of decision-making and memory (3.11), and they compound over long tasks: a small chance of a wrong step at each stage becomes a large chance of failure across a task with many stages.
Why it matters: Effort now pays off most in choosing the next step, in memory, and in reliability over long tasks.
← Part 2. The industry in numbers, 2026 Part 4. Data, simulation and world models →