Physical AI glossary
What is Benchmark?
In the guide: 2.4 Imitation learning and VLAs went mainstream
The debate: The report argues that beyond roughly 7B parameters, curation beats scale: a well-curated 500-demonstration fine-tune of a 7B model outperforms a poorly curated fine-tune of a 70B model on most manipulation benchmarks. If it holds, competitive advantage lies in collecting and cleaning task data rather than in training ever larger models, though benchmark results do not always carry over to real deployments.
In the guide: 4.10 Evaluation
- A three-stage check before deployment. First, set targets for accuracy and time (if one bus bar takes a day, no deployment wants it). Second, work through a list of disruptions, such as a box left in the way or the lights going off, to measure how the robot behaves outside the conditions it was trained in. Third, test safety rules, such as stopping when someone cuts the camera cable, to confirm that faults lead to a safe state.
- Benchmarks based on paid tasks. Use customers' real tasks and parts, so that passing the benchmark means the robot is commercially useful rather than merely competent on a laboratory proxy.
- Throughput, not only success rate. Measures such as boxes per hour combine success and speed in a single figure, and companion measures such as interventions per hour and mean time between failures (MTBF) capture how often a human must step in and how long the robot runs unaided.
- The customer's standard, not the researcher's. A restaurant grades a folded napkin on a scale where acceptable and unacceptable can differ by about an inch in a single fold, a distinction a binary research label of success or failure would miss.
In the guide: 4.11 What the benchmarks show
Most robot results are measured on a team's own tasks, so two results are rarely comparable. A shared benchmark runs every model on the same tasks under the same scoring. One of the most complete in 2026 is RoboDojo: 42 manipulation tasks in simulation and 18 on real robots. Instead of one success rate, it scores five separate abilities: