Every millisecond counts

Our robots run their control loops at up to 100 Hz: every 10–20 milliseconds the system reads joint states and camera frames and sends the next command. Late data has physical consequences — a jerk in the arm, a missed grasp, an episode that cannot be used for training. With the larger and more powerful models, moving the inference on cloud adds additional challenge to the latency budget. In this blog we describe how the stack is built around this constraint, from data collection to inference including running the inference on Tenstorrent stack with custom kernels for maximum performance.

The fundament for the control loop is the hardware. Leader arms are polled at the control rate over a serial bus, and follower arms are driven over 1 Mbit CAN with PD gains tuned for the 100 Hz loop. Serial adapters batch reads with a 16 ms latency timer, which consumes most of a control tick inside the kernel; a udev rule sets it to 1 ms on every station. Cameras run at 60 fps and every frame carries an age budget of two frame periods — a frame older than that is treated as a fault, not as data.

We keep the compute intense work out of the loop. Dataset writes and video encoding run in a separate recorder process. Frames are passed to it through a shared-memory slot pool without serialization; the queue between the processes carries only slot indices. Episodes are synced out nightly and pass through the quality filters described in an earlier post.

Cores on the deployment platform are partitioned: the operating system and the time-critical processes — the control loop, inference, the recorder — each run on their own set, and the host is tuned end to end so nothing preempts the loop, from scheduling to memory management to the ML runtime. One device is one process, and processes are crash-only. Supervision is delegated to systemd: each driver pets a watchdog on every published sample, so a stalled or unplugged camera starves the watchdog and is restarted within seconds while the rest of the system keeps running.

Within a host, processes communicate over iceoryx2, a zero-copy shared-memory transport. Its performance does not depend on the number of subscribers, so attaching a viewer, a topology observer, or a debug recorder to a camera stream does not affect the control loop. This is also what lets the loop coexist with the processes and services built around it.

We recently ported our model inference onto a Tenstorrent stack. The model runs on tt-metal with the compute graph captured as a device trace, and two command queues overlap the work: while one replays the graph for the current request, the other stages the next observation's images and tokens onto the card. We have optimized the performance with the largest single improvement coming from restructuring the vision-language prefill to process all three camera views and the prompt in one pass across the chip:

Served end-to-end over gRPC, the round trip is 70.6 ms at the median, and 71.2 ms at the maximum. The worst case is what matters for a real-time system. For context, a fifty-action chunk at 50 Hz corresponds to 1 second of motion, so a full round trip costs 5.5% of one chunk.

Reaching a flat tail meant fixing several issue specific to real-time serving. The captured trace initially replayed against whatever was in the device's input buffers, so every chunk was computed from the previous observation; the ordering is now covered by a regression test. Early runs spiked to 102 ms at p90 because the host rebuilt an 8 MB positional-encoding table twice per request only to index a fraction of it; caching the table brought the maximum from 225 ms to 65.7 ms. A changed image-resize implementation made preprocessing jump from 3 ms to 140 ms because the tensor library's default thread pool contended with the device runtime; limiting it to one thread brought it back to 2.8 ms.

The control loop and the policy are separate processes connected by gRPC through a single address setting. The same code runs with the policy on the robot's host or on a separate machine with accelerators. Robots, workstations, and inference machines share a private mesh network that we control end to end; a round trip between a robot station and an inference machine measures 0.7 ms on average. The endpoint itself is unauthenticated — the network is the trust boundary — and the server refuses to bind a wildcard address.

The protocol keeps the loop off the network path. Observations are sent without waiting for a response, action chunks come back on a long-lived stream, and both directions hold only the latest message: backlog is dropped, since actions computed against a stale observation are useless. Keepalives detect a half-open peer within seconds.

During operation we need to know what failed, where, and what state the system is in. Everything that leaves a host travel over a pub/sub protocol with low enough overhead to run next to the control loop, pinned to a different core. Each component — the loop itself, every camera, the robot thread, the recorder, the policy client — publishes health on its own topic: observed rates, overrun counts, frame ages, inference latency. The run state is published with last-sample replay, so a dashboard attached in the middle of an episode sees the current state immediately. Camera previews are JPEG-encoded on a worker thread and published on separate topics so an operator subscribed to the same topics remotely, can follow an episode without touching the loop.

There is one supported way to run the stack — the production way — with development ergonomics provided by tooling rather than a parallel code path, and a deliberately short dependency list. A new robot platform, camera, or inference backend is one process replaced, publishing the same messages. These processes let a robot work through a shift in a live warehouse, the data engine run around the clock, and a new accelerator enter the stack in days.

If you are interested in building this kind of infrastructure for physical AI, see our open positions or reach out at contact@lute.one.