The narrative around AI robotics usually skips the boring middle. You hear about the model and you hear about the demo, but the ninety percent in between — the loop that turns pixels into motor commands, reliably, thousands of times a day — is where teams actually spend their time.
The loop that matters
A production autonomy stack is really four systems pretending to be one:
- Perception turns raw sensor data into a model of the world.
- Planning decides what to do given that model.
- Control turns decisions into smooth, safe motion.
- Ops keeps all of the above healthy across a whole fleet.
A great model helps perception. It does almost nothing for the other three.
The hardest bug we ever shipped wasn't in the model. It was a 40ms scheduling jitter that only appeared when the battery dropped below 20%.
What we optimize for
When we evaluate a change to the stack, we care about three numbers:
| Metric | What it tells us | Target |
|---|---|---|
| Loop latency | End-to-end perception → control time | < 50ms |
| Intervention rate | Human takeovers per autonomous hour | Trending 0 |
| Recovery time | Time to safe state after a fault | < 200ms |
Notice that model accuracy isn't on the list. Accuracy is a means, not an end — what the operator feels is latency and reliability.
A tiny example
Here's the shape of a control tick. It's deliberately unglamorous:
function tick(state: RobotState): Command {
const world = perception.update(state.sensors);
const plan = planner.next(world, state.goal);
return controller.follow(plan, state.dynamics);
}
The magic isn't in any one line. It's that this function runs on time, every time, even when a sensor drops out or a wheel slips.
Where to start
If you're building your first robot, resist the urge to start with the model. Start with the loop. Get a boring, reliable teleop pipeline working end to end, then replace the human piece by piece. That's the path we walk through in the tutorials.