Success rate vs. reliability: why 90 percent is not enough
A robot that succeeds 90 percent of the time gets through ten attempts in a row only about 35 percent of the time. Demos show the first number.
A robot that succeeds on 90 percent of its attempts will still fail most long runs. Every attempt is a fresh chance to fail, so the odds multiply, and at 90 percent per attempt the robot gets through ten attempts in a row without a failure only about 35 percent of the time.
Two numbers that sound like the same thing
Success rate is the share of individual attempts that work. The robot picks up a part and sets it in a fixture, and you count how often it lands over a hundred tries.
Reliability is the chance that the system gets through a long stretch of work, a whole batch or a whole shift, without failing.
People mix these up because both are percentages and both seem to answer "how good is the robot." Work on a production floor is many attempts strung together, and one failure anywhere in the string stops it, so a robot can post a respectable success rate and still be unreliable.
General robot models, meaning learned models built to handle many different tasks, usually get 80 to 90 percent of their attempts right once they're deployed.
The arithmetic of a long run
If each attempt succeeds or fails on its own, the chance of a clean run is the per-attempt rate multiplied by itself once for every attempt. At 90 percent, ten in a row comes out at about 35 percent. At 80 percent, it's about 11 percent.
So most ten-attempt runs will need a person to step in at least once, and a shift holds far more than ten attempts.
If you want a hundred actions in a row with a fair chance that none of them fails, each attempt has to succeed well over 99 percent of the time. The distance from 99 percent to 99.99 percent is also bigger than it looks. Each extra nine means getting rid of most of the failures that were still left at the level before, and the ones that survive tend to be the rarest and strangest cases. Those are the hardest to train for, because rare situations barely show up in the data.
Failures come in clusters
The multiplication treats every attempt like a separate coin flip, while failures on the floor cluster. If a robot has trouble with one unusual part, it will probably have trouble with every part like it. A change in lighting can make performance sag for the whole shift.
Clustering like this can push reliability on the floor above or below what the multiplication says, and a short test won't tell you which way. So people who run robots measure reliability straight off long runs.
Mean time between failures (MTBF) is how long, on average, the system runs before something goes wrong. Interventions per hour counts how often a person has to step in and correct or reset the robot (turn it upside down and you get mean time between interventions). Both track what a plant actually experiences far better than a per-attempt success rate does.
Why fixed automation is so reliable
Classic industrial arms can be extremely reliable. A fixed arm running a hard-coded path inside a guarded cell sees parts arrive in the same place and the same orientation every time, because the cell was engineered to strip out variation.
That reliability costs flexibility: a new task, or a part that shows up slightly out of position, usually means new fixtures and new engineering. Learned robots sit at the other end. They cope with variation and can take on many tasks, but they haven't reached that level of reliability yet. Some startups are trying to remove the trade-off with learned models that stay flexible and are still dependable enough to sell. Getting from a robot that works in a demo to one that works nearly every time is the hardest open problem in robotics right now.
How to read a robot demo
Robot videos almost always show the runs that worked. That makes a video a best-case success rate, and it tells you nothing about reliability. When someone shows you one, ask:
- How long did it run, start to finish?
- How many attempts were made, and how many of them failed?
- Were any failures edited out?
- How often did a person step in, and how long did it typically run between failures?
A short clip with no answers to these shows the robot is able to do the task. Whether it can do the task all shift is a separate claim, and only long, measured runs back that one up. For how lab tests and benchmarks are set up, see how to evaluate a robot.
The arithmetic above also assumes that a single failed attempt ends the task. In practice a capable robot, like a capable worker, often notices the miss and tries again. That ability is the biggest lever on reliability, and how robots recover from mistakes explains how robots learn it.
Go deeper in Physical AI, Explained
This post draws on the chapter From demo to reliable product of our free guide to how robot models, data, hardware and deployment fit together.
Read the chapter