# Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs

arXiv: https://arxiv.org/abs/2412.04372 · HTML: https://mlsanity.com/paper/2412.04372/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2024.

## Built on by (most cited first, 3 of 3)

- [Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation](https://mlsanity.com/paper/2605.14047/) (2026)
- [Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices](https://mlsanity.com/paper/2605.15694/) (2026)
- [Recursive transformers for semiconductor thermo-mechanical reliability](https://mlsanity.com/paper/2607.27251/) (2026)

## Abstract (arXiv)

Contextual Artificial Intelligence (AI) based on emerging Transformer models is predicted to drive the next technology revolution in interactive wearable devices such as new-generation smart glasses. By coupling numerous sensors with small, low-power Micro-Controller Units (MCUs), these devices will enable on-device intelligence and sensor control. A major bottleneck in this class of systems is the small amount of on-chip memory available in the MCUs. In this paper, we propose a methodology to deploy real-world Transformers on low-power wearable devices with minimal off-chip traffic exploiting a distributed system of MCUs, partitioning inference across multiple devices and enabling execution with stationary on-chip weights. We validate the scheme by deploying the TinyLlama-42M decoder-only model on a system of 8 parallel ultra-low-power MCUs. The distributed system achieves an energy consumption of 0.64 mJ, a latency of 0.54 ms per inference, a super-linear speedup of 26.1 x, and an Energy Delay Product (EDP) improvement of 27.2 x, compared to a single-chip system. On MobileBERT, the distributed system's runtime is 38.8 ms, with a super-linear 4.7 x speedup when using 4 MCUs compared to a single-chip system.