Physical AI, Explained · Part 9
Compute and supply chain
Read this part in the book →Chips and supply chains set the floor on the cost and risk of everything else a robotics company does. Inference hardware is a recurring cost on every robot, training compute is a large fixed cost on every model, and the availability of motors, magnets and sensors determines whether robots can be built at all. A founder therefore needs the basics even without being a chip analyst.
- Training and inference. Training is the process of building a model: adjusting its parameters over very large datasets, which requires large clusters of chips running for days to months. It is a large cost, incurred once per model version. Inference is the process of running a trained model to produce outputs, such as the next motor commands for a robot. Each inference step is cheap, but it is repeated continuously for as long as the model is in use. A fleet of deployed robots runs inference all day, so inference cost scales with the number of robots and hours worked, while training cost scales with the number of models built. The two workloads stress hardware differently: training needs enormous memory bandwidth and fast links between many chips, while inference on a robot needs low latency (the delay between an observation and the resulting action) within tight power and heat limits. Chips are increasingly specialised along the same line, with some designed for training and others for inference.
- GPUs, ASICs, and stranded investment. GPUs (graphics processing units) are general-purpose parallel processors that have become the standard chips for AI. One maker dominates the market, largely because of its software ecosystem: the programming tools and optimised libraries that developers already know make switching to another vendor costly even when competing hardware is comparable. ASICs (application-specific integrated circuits) are custom chips designed for one workload. Because they omit circuitry that the target workload does not need, they cost less per unit of work and use less power, but they carry a specific risk: if model architectures change, the chip may no longer run the new models efficiently, and the money spent designing and buying it is stranded. The same reasoning is one argument for keeping expensive computing off the robot, where it is fixed for the robot's lifetime, and placing it in servers that can be upgraded or repurposed, at the cost of depending on a network connection.
- Distillation. Distillation is a technique in which a large, capable model (the teacher) is trained first, and a smaller model (the student) is then trained to reproduce the teacher's outputs. The student learns from the teacher's full output distributions or actions rather than only from the original labels, which carries more information per example and lets a much smaller model retain much of the teacher's capability. The student is fast and cheap enough to run on the robot. Robotics is likely to rely on distillation widely, because a robot's control loop must produce actions within strict time limits that a very large model cannot meet on onboard hardware.
- Where the gains come from. Most of the improvement in intelligence per watt in recent years came from better models and algorithms, such as more efficient architectures, better training methods and lower-precision arithmetic, not from hardware alone, though one hardware generation did deliver about 30x. Algorithmic progress spreads quickly through publications and open models, so a focused startup is better placed to build on other companies' model progress than to try to outspend them on it.
- Not yet a bubble. Demand for valuable AI work is still growing faster than computing capacity can be built, which is why computing prices are rising rather than falling. A bubble would show the opposite pattern: capacity outrunning paying demand, and prices falling as a result.
- Co-design. Co-design means designing hardware, models and system software together, so that each is shaped around the others. Gains then multiply rather than add, because an improvement in one layer can unlock improvements in the others that would not be possible in isolation. A model designed to suit a chip's memory layout, for example, may allow a compiler optimisation that would otherwise be impossible. Three 2x improvements designed together can approach 100x instead of 8x. This does not conflict with building a product that can switch models (Part 10), because the two operate at different layers: co-design happens within the model and inference stack, while model-agnosticism is a property of the product architecture above it.
- Supply chain. Actuators, the permanent magnets inside their motors, and many sensors depend on production concentrated in a few countries, with rare-earth magnet materials an especially concentrated case. Moving production back home is slower and harder than it looks (5.8), because it requires not only factories but also the surrounding network of material processors, specialised suppliers and experienced workers that took decades to build elsewhere.
Why it matters: Two of these factors, inference cost per robot and dependence on Chinese supply chains, show up directly in a robotics company's margins and risk.
← Part 8. Markets and capital Part 10. What the robotics market rewards →