Physical AI glossary
What is VLA model?
In the guide: 2.4 Imitation learning and VLAs went mainstream
VLA models (vision-language-action models) are large neural networks that take camera images and a natural-language instruction as input and output robot actions, typically built by extending a model pretrained on internet-scale image and text data. Almost absent from production in 2024, they now underpin about 40% of new deployments, according to the report. The report also finds that fine-tuning a base VLA, meaning continuing to train an already pretrained model on a small task-specific dataset, on 200 to 500 task demonstrations now outperforms training a task-specific policy from scratch on 1,000 or more. The mechanism is transfer: the pretrained model already encodes general knowledge of objects, scenes and language, so the fine-tuning data only has to teach the specifics of the new task and robot. The comparison depends on the tasks and benchmarks chosen, and the report's summary does not allow it to be checked independently.
In the guide: 3.5 Three families of robot model
1. Vision-language-action models (VLAs). A VLA is a large language model adapted to control a robot. It is built in three stages: 1. Train it on text, so it learns language and general knowledge. 2. Train it further on web images paired with text, so it learns to connect words to what it sees. A model at this stage is called a vision-language model. 3. Train it further on varied robot data, so it learns to output actions instead of words.
In the guide: 4.9 Simulation spectrum and world models
The current role: World models used to run only offline, for development and testing, and some now run inside the robot's live decisions. One recent vision-language-action model (VLA) uses a world model while it works, to picture where the robot should be a second or two ahead, and then chooses actions that move it toward that predicted state. World action models predict future video as part of choosing their actions (3.5). A robot has also been shown imagining training scenarios inside its own model, described as the robot dreaming, to extract more learning from less than an hour of real data; the underlying idea, training a policy on experience generated by a learned model rather than by the real world, is long established in model-based reinforcement learning. Two cautions apply. The imagined images help perhaps less than one might expect, because errors in the model become errors in the training data, and enthusiasm for world models can distract from the underlying constraint, which is still data.