Physical AI glossary
What is Sim-to-real gap?
In the guide: 3.7 Imitation vs. reinforcement learning
RL has its own difficulties. The first is reward design. A sparse reward, given only when the task succeeds, is easy to specify but gives the robot almost no information during the many early attempts that fail. A dense reward, which scores partial progress such as distance from the gripper to the object, speeds learning but can be exploited: the robot may find a way to increase the score without doing the task, a behaviour called reward hacking. A common modern approach is to train a separate model to judge from camera images whether a task has succeeded or how much progress has been made, and to use its judgement as the reward. The second difficulty is sample efficiency: RL typically needs very many attempts, which is cheap in simulation and slow, expensive and potentially damaging on real hardware, where a person may also have to reset the scene after each attempt. The third is safe exploration, since trying random movements near people or fragile objects is unacceptable. For these reasons much RL is done in simulation, with domain randomisation (varying friction, mass, lighting, sensor noise and other simulator parameters during training so that the real world looks like just another variation) used to make the learned behaviour survive the transfer to a physical robot, the sim-to-real gap.
In the guide: 3.12 Data and scaling laws
Data can be scarce in two distinct ways. In some cases the data does not exist and has to be generated, through teleoperation, simulation or the robot's own practice. In others it is being produced somewhere, for example in the motions of workers or in the logs of existing machines, but nobody has built a way to capture it in a form a model can learn from. Each source has a characteristic weakness. Teleoperation is accurate but slow and labour-intensive. Simulation is cheap and unlimited but suffers from the sim-to-real gap (3.7), which is widest for contact-rich manipulation. Video of people is abundant but contains no motor commands or forces and shows human hands, not robot grippers. Most serious efforts therefore combine several sources.
In the guide: 4.8 Simulation and sim-to-real
Simulation means training or testing a robot in a virtual world governed by a physics engine, software that computes how bodies move, collide and exert forces on one another at each small time step. It is used because real-world data is slow, expensive and sometimes dangerous to collect, while a simulator can run many robots in parallel, faster than real time, and reset instantly after a fall. The sim-to-real gap, also called the reality gap, is the difference between the simulator and reality, which means a skill learned in simulation may fail on the real robot. Its main sources are well known: contact and friction are hard to model accurately, motors and gears have delays, backlash and saturation that simulators simplify, rendered images differ from real camera images, and soft or deformable objects such as cloth and cables are expensive to simulate at all. A perfect simulator would amount to having roughly solved the universe, so the gap never fully closes.