Physical AI glossary
What is Foundation model?
In the guide: 3.3 The case for general models
The leading research groups are betting on robot foundation models. A foundation model is a single large model, trained on broad data, that can be adapted to many downstream uses; in robotics, the aim is one model that controls many kinds of robots and performs many tasks, instead of a separate system engineered for each. The idea comes from language processing. Researchers first built narrow systems, one for translation, another for sentiment analysis, another for question answering, each trained on its own curated dataset. General language models trained on vast amounts of text then outperformed all of them. Training on everything produced a broad internal representation of language and the world on which specific skills could build.
In the guide: 3.12 Data and scaling laws
Data is the limit, for four reasons: - The gap is very large. One estimate puts the data a robot foundation model needs at around 100 trillion examples. The largest robot teleoperation datasets hold around a million, so almost the whole 100 trillion is missing. The figures are one company's estimate, and the conclusion holds even if they are far off. - More data does not mean more variety. In language, more data brings more variety automatically, because the internet covers so much of human life. In robotics it does not. Trillions of hours of self-driving data are still all driving. A hundred robots in one room learn only about that room, and 100 trillion tokens are useless if they all come from the same environment. - Variety matters more than volume, and this has been measured. When the most varied data was removed from one training set, performance on new tasks dropped sharply. Removing a random fifth of the less varied data barely changed anything. - Even unrelated data helps. Adding data unrelated to a task, even data collected in another country, has been reported to improve performance on it, with no sign of the effect levelling off.
In the guide: 4.3 Teleoperation
- It cannot produce enough data for a foundation model on its own. Each demonstration takes a few minutes of a person's time and a robot's time. Even millions of operators would need an unrealistic amount of time to approach the volume of data on which language models are trained, so teleoperation is better suited to targeted, high-value data than to bulk pre-training.
- The robot can only be as good as the operator. A model trained purely to imitate human demonstrations learns, at best, the demonstrator's skill, and teleoperated skill is lower than the same person's skill with their own hands because the interface removes touch and adds delay. Some behaviours cannot be teleoperated at all. Balancing and running require reactions within milliseconds to forces the operator cannot feel, so legged locomotion is learned by other means, chiefly simulation (4.8).
- It is slow. People move more slowly and cautiously when controlling a robot remotely, because of latency, a limited camera view and the absence of touch. Robots trained on that data inherit the slowness. Teams either make the data faster, for example by better interfaces or by speeding up recorded trajectories, or train the robot to execute faster than the demonstrations it learned from.
- The work has to happen where the robot is. Each session needs a working robot, power, network and setup on site, so the data comes from only a few places and covers a narrow range of environments.