Polymorph/Physical AI, ExplainedGlossary

Physical AI, Explained · Part 6

From demo to reliable product

Read this part in the book →

A robot that works most of the time makes a good video. A business needs one that works nearly every time, and closing that gap is the hardest problem in the field today. This part explains why the gap exists, and the methods practitioners use to narrow it: teaching robots to recover from their own errors, training models that measure progress, making reinforcement learning cheap enough to run on physical hardware, directing human effort at the cases where the robot is weakest, extending autonomy from single tasks to multi-hour workflows, and raising speed.

6.1 The gap between demos and reliability

Producing robot behaviour that looks impressive is no longer the main difficulty. Washing a pan, peeling vegetables and making a grilled cheese sandwich are all achievable with current learned models. What remains difficult is turning such a capability into a system that delivers value to someone who depends on it, shift after shift, without an engineer standing by.

The distinction is between success rate, the fraction of individual attempts at a task that end in success, and reliability, the probability that a system completes a long run of work without failing. A demonstration shows the first. A deployment depends on the second. General robot models typically succeed 80 to 90 percent of the time per attempt in real deployments, and because each attempt is a separate opportunity to fail, those probabilities multiply. At 90 percent per attempt, the chance of ten consecutive attempts without a failure is 90 percent multiplied by itself ten times, about 35 percent. At 80 percent it is about 11 percent. To complete a hundred actions in a row with reasonable confidence, the per-attempt success rate must be well above 99 percent, and 99 percent is a long way from 99.99 percent: each additional "nine" of reliability requires eliminating most of the failures that remained at the previous level, and those remaining failures are usually the rarest and strangest cases, which are the hardest to collect data for.

The multiplication assumes that attempts fail independently, which is a simplification. In practice, failures are often correlated: a robot that struggles with one unusual object tends to struggle with every object like it, and a change in lighting can depress performance for an entire shift. Correlation can make real-world reliability either better or worse than the simple calculation suggests, which is why practitioners measure it directly over long runs instead of extrapolating from short tests. Common operational metrics include mean time between failures (MTBF), the average duration a system runs before a failure occurs, and interventions per hour (or its inverse, mean time between interventions), which counts how often a human has to step in to correct, reset or rescue the robot. These numbers capture what a customer experiences far better than a per-attempt success rate does.

Classic specialist machines, such as a fixed industrial arm executing a hard-coded trajectory inside a guarded cell, can be extremely reliable because their environment is engineered to remove variation: parts arrive in the same place, in the same orientation, every time. The cost of that reliability is inflexibility. Reprogramming the machine for a new task, or tolerating a part that arrives slightly out of position, typically requires new fixtures and new engineering. Some startups aim to remove this trade-off by building learned models that can perform many tasks and are also reliable enough to sell.

Why it matters: When evaluating a demo, ask how long it ran, how many attempts were made, and how many failures were edited out. Robot videos typically show only the successful cases, so they report a best-case success rate and say nothing about reliability.

6.2 Recovering from mistakes

The compounding calculation in 6.1 assumes that one failed attempt ends the task. That assumption is often wrong, and the reason it is wrong is the most important lever on reliability. People are not especially accurate either. A warehouse worker makes far more than one wrong pick in a million, yet still finishes almost every order, because after a mispick the worker notices and picks again. Completion rate depends on per-attempt accuracy and on the ability to detect and correct errors, and the second factor can dominate. Robot models are beginning to show the same pattern: recovering from mistakes raises the task completion rate well above what per-attempt accuracy alone would predict.

A standard model trained purely by imitation shows the opposite behaviour. It performs well until it makes a small mistake, then typically goes out of distribution, meaning it faces a situation unlike anything in its training data, and from there it gets stuck or makes things worse. The mechanism is well understood. In imitation learning, also called behaviour cloning, a policy is trained to copy the actions in human demonstrations, and those demonstrations almost always show an expert doing the task correctly. The model therefore never sees what to do after an error. Once a small deviation occurs, the robot is in a state the demonstrations did not cover, its next action is less accurate, which pushes it further from familiar states, and the errors accumulate. This is called compounding error, and the mismatch between the states seen in training and the states the robot actually visits is called covariate shift. The standard remedy, associated with an algorithm known as dataset aggregation, is to let the robot run its own policy, record the states it drifts into, and have a human supply the correct action in those states, so that the training data comes to include the situations the robot itself creates.

In one reported case, the fix was to teach the robot explicitly how to recover, and that skill turned out to generalise beyond the specific recoveries it was shown. After enough training rounds, the robot knocked over a whole stack of napkins, separated them, and carried on with the task, even though it had never been shown how to handle that particular accident.

This broader ability can be called improvisational intelligence: producing sensible behaviour in situations nobody planned for. In older robotics, every behaviour was a hand-built controller, often a state machine with explicit branches for each anticipated condition, and it worked only in the situations its designers had foreseen. An unanticipated condition fell through every branch and the system halted or did something wrong. A learned model, shown something new, will usually do something reasonable, because it has learned general regularities about objects and actions instead of a list of cases. It does not always succeed, but it fails gracefully more often, and graceful failure is what makes recovery possible.

6.3 Progress meters: reward models and value functions

Once a robot succeeds around 90 percent of the time, paying people to watch every run in order to catch the rare failure becomes wasteful, and human attention becomes the bottleneck on further improvement. The solution, which two groups arrived at independently, is to train a separate model that watches the robot and scores how well it is doing. Such a model turns hours of unlabelled robot footage into a signal that says where things went right and where they went wrong.

The value function is the more powerful of the two because it supports a precise judgement about individual actions. If the estimated time remaining falls after an action, the action helped. If it rises, the action hurt. The difference between what an action achieved and what the value function expected is called the advantage, and it is the quantity most reinforcement learning methods use to decide which behaviours to strengthen and which to suppress. Both kinds of model share a known weakness: a learned scorer can be wrong in ways the policy learns to exploit, scoring an attempt highly because it superficially resembles success. Practitioners guard against this by checking the scorer against human judgement on held-out episodes and by retraining it as the robot's behaviour changes.

Why it matters: Progress meters replace continuous human supervision with automatic scoring, which makes it affordable to keep improving robots after deployment.

6.4 Making real-world RL affordable

Reinforcement learning (RL) is the family of methods in which an agent improves by trying actions, observing the results, and adjusting its policy toward actions that led to better outcomes. Labs improve language models with RL over millions of attempts, which is cheap because each attempt is only computation. For robots, every attempt happens in physical time on physical hardware. One million attempts at a one-minute task takes about 700 robot-days, and the hardware wears, objects must be reset between attempts, and some failures damage things. Language-model RL also tries each prompt 10 to 50 times simply to estimate which answers are better than average. On real hardware, that sampling is far too slow.

The property at stake is sample efficiency: how much improvement a method extracts per attempt. Two broad strategies raise it. Offline RL learns from a fixed dataset of previously recorded experience, including other policies' attempts and human demonstrations, without collecting anything new. Online RL collects fresh experience with the current policy and learns from it as it goes, which corrects the policy's actual mistakes but costs robot time. Practical robot systems usually combine the two, extracting as much as possible from existing data and spending real-world attempts only where they add information. One process makes this practical:

  1. Train a foundation model on varied data.
  2. Deploy it, with a person stepping in to stop it from wasting time on dead ends. For example, if the robot picks up two cardboard boxes stuck together and keeps trying to fold them as one, a person slides a finger between them and lets the robot carry on. These interventions are cheap, keep each attempt productive, and themselves become training data showing the right correction at the moment it was needed.
  3. Train a general value function on that experience.
  4. Use the value function to improve the robot's policy, meaning the way it decides what to do, by favouring actions the value function judges to have moved the task forward and disfavouring those that set it back.
  5. Repeat.

Each cycle produces a better policy, whose deployment produces better data, which trains a better value function. Reported results include roughly double the throughput from the RL stage alone, over 90 percent success at making espresso, a latte service that ran for 13 hours straight, and a real box-assembly job at a customer's factory, a job the workers there disliked. These results are still not 99.99 percent reliable, and the robots are still slower than people.

One unusual theory addresses why RL works at all in very large models. A research paper found that in such models, random changes to the weights often still produce good results, so RL may work partly because it adds random noise that explores nearby behaviours, not because each update does anything especially smart. This remains a hypothesis about mechanism and does not change the practical recipe above.

6.5 Human-in-the-loop active learning

A third route to reliability sits between collecting more demonstrations and letting the robot practise alone. The robot runs on the real task, the reward model shows where it struggles, people record demonstrations of how to recover in exactly those situations, the model is retrained, and the cycle repeats. The model's own weaknesses decide what data to collect next, which is what active learning means: instead of gathering data uniformly, the learner directs collection toward the examples it is most uncertain about or most likely to get wrong. Uniform collection spends most of its effort on situations the model already handles, while targeted collection spends it on the failure modes that limit reliability.

On restaurant napkin folding, this raised success from about 80 percent with standard training to 99.4 percent over a 24-hour run (a reported result), and the robot was not thrown off by morning light changing through the windows. Gradual lighting change is a classic source of covariate shift for vision-based policies, so holding up through it indicates that the corrective data covered real variation in the environment. The process resembles reinforcement learning from human feedback (RLHF), the method used to tune chatbots, in which people rate a model's outputs and those ratings are used to train it. The robot tries, people grade the attempts, and both the grades and the new demonstrations train the model.

Why it matters: For a company deploying robots today, this is the most practical way to improve reliability, because it runs on the customer's actual task and spends human effort only on the cases that are failing.

6.6 Workflows that run for hours

Most robot autonomy is still measured in tasks of 30 seconds to a minute, while the commercial value lies in workflows that run unsupervised for hours. A long workflow is a sequence of many tasks, so the compounding arithmetic of 5.1 applies with full force, and it adds problems a single task does not have: deciding what to do next, waiting on other machines, and remembering what has already been done.

Getting there takes more than a better low-level model:

6.7 Speed

Robots are still slower than people, partly because teleoperated demonstrations are slow. A person controlling a robot remotely sees through cameras, lacks a full sense of touch, and moves cautiously, and a policy trained by imitation reproduces that caution, including the pauses. RL helps remove such hesitation. A person may pause to line up a cable plug, for instance, but a policy optimised for completing the task quickly can learn that the pause is unnecessary and skip it. One later model ran faster than human teleoperation, and one napkin-folding robot reached human speed without slowing from fatigue (both reported results).

The usual measure is throughput, the number of units of work completed per hour, which combines speed and reliability: a fast robot that needs frequent resets can have lower throughput than a slower one that never stops.

Why it matters: Customers pay for output per hour. A reliable robot that works at half a person's speed often does not pay for itself.


← Part 5. Hardware   Part 7. Deployment and go-to-market →