A deep dive into on-device vs. data center inference for robots, including a primer on robot models, deployments, supply chains, the "network wall", and more

If you build robots, the cost and performance of onboard processors will improve less quickly than datacenter GPUs.

The article explores the optimal location for a robot's artificial intelligence: on-device versus in a data center. Unlike traditional AI, which prioritizes model performance, robotics faces unique constraints. Robots require real-time control, meaning delays can render actions obsolete. Furthermore, the cost of embedded compute must be factored into each unit, potentially leading to massive upfront capital expenditure at scale. Consequently, robot models are typically smaller than LLMs, designed to fit within the hardware and latency budgets of on-board systems. While current generalist robot models range from 3 to 10 billion parameters, their size is dictated by available hardware like NVIDIA's Jetson or H100. The article notes that NVIDIA's DreamZero, a 14-billion-parameter model, requires significant off-robot compute, while RoboTTT uses a smaller, self-updating 3-billion-parameter model designed for onboard execution. The supply chain also favors data center silicon over robot-specific chips, with NVIDIA prioritizing high-margin data center GPUs.

The physical constraints of robotics, specifically the need for real-time execution and the high cost of embedded hardware, mean that robot models are designed to fit available compute rather than pushing model size limits. This contrasts sharply with LLMs, where model size is optimized for quality and inference cost, leading to a divergence in hardware requirements. The choice between on-device and data center inference for robots will therefore be a complex trade-off between latency, cost, and the computational demands of increasingly generalist models.

The supply chain further complicates matters, as manufacturers like NVIDIA prioritize high-margin data center silicon over the nascent robotics market. This means advanced process nodes and manufacturing capacity are directed towards datacenter GPUs, leaving robot-specific hardware, like NVIDIA's Jetson line, with less cutting-edge technology and potentially higher costs in the future. As robot models grow in complexity, their reliance on data center compute will increase, but overcoming the "network wall" of latency and jitter for real-time control remains a significant engineering challenge.