Physical AI glossary
What is Distillation?
Distillation: training a small model to imitate a large one, so it can run on the robot.
In the guide: 11.1 What each trend makes cheap
- Models. One general model now matches its maker's fine-tuned specialists with no extra training (a reported result). A fine-tuned specialist is a copy of a model given additional training on one narrow task; when a single general model performs as well without that step, the specialist's extra engineering stops being an advantage. Strong robot foundation models, large models pretrained on broad robot and internet data and then adapted to particular robots and tasks, are already released as open source. Techniques also spread quickly: once a group publishes early results, the method, the architecture and often the training recipe can be reproduced, and others catch up within months. Any lead built on technique alone is therefore temporary. General robot intelligence appears to be following the path general language intelligence took, improving every year and becoming available to everyone.
- Compute. Each new generation of accelerator chips performs more operations per unit of cost and energy, and each algorithmic improvement (more efficient architectures, lower-precision arithmetic, better training schedules) reduces the computation a given capability needs. Together these lower the cost of running a model of fixed capability year after year. Distillation, training a small "student" model to imitate the outputs of a large "teacher", makes it practical to train a large model in a data centre and then run a compact version on the robot's onboard computer, where power, heat and latency limits are strict. Total demand for computation is still rising faster than supply, so the price of the most advanced capacity stays high, but the cost of any particular level of capability keeps falling.
- Components. Fourteen manufacturers sell arms under $10,000, a dozen humanoids are available to buy or lease, and leader-follower rigs cost under $2,000 (RCSV figures). A leader-follower rig is a teleoperation setup in which a person moves a lightweight "leader" arm by hand and a powered "follower" arm copies its joint positions, which is the standard way to record demonstration data cheaply. Chinese manufacturing, with its dense supplier networks for motors, gearboxes, sensors and electronics, pushes these prices down further.