# Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices

arXiv: https://arxiv.org/abs/2605.15694 · HTML: https://mlsanity.com/paper/2605.15694/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 15 May 2026.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [Distributed Deep Convolutional Neural Networks for the Internet-of-Things](https://mlsanity.com/paper/1908.01656/) (2019)
- [DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices](https://mlsanity.com/paper/2309.05015/) (2023)
- [Communication-Efficient Distributed On-Device LLM Inference Over Wireless Networks](https://mlsanity.com/paper/2503.14882/) (2025)
- [Communication-oriented Model Fine-tuning for Packet-loss Resilient Distributed Inference under Highly Lossy IoT Networks](https://mlsanity.com/paper/2112.09407/) (2021)
- [Integrating Large Language Models with Internet of Things Applications](https://mlsanity.com/paper/2410.19223/) (2024)
- [Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference](https://mlsanity.com/paper/2410.11650/) (2024)
- [Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality with At-MRAM Neural Engine](https://mlsanity.com/paper/2312.14750/) (2023)
- [Synchronous Transmissions in Low-Power Wireless: A Survey of Communication Protocols and Network Services](https://mlsanity.com/paper/2001.08557/) (2020)
- [Wireless Control for Smart Manufacturing: Recent Approaches and Open Challenges](https://mlsanity.com/paper/2010.09087/) (2020)
- [Restructuring, Pruning, and Adjustment of Deep Models for Parallel Distributed Inference](https://mlsanity.com/paper/2008.08289/) (2020)
- [Optimization Framework for Splitting DNN Inference Jobs over Computing Networks](https://mlsanity.com/paper/2111.07006/) (2021)
- [Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs](https://mlsanity.com/paper/2412.04372/) (2024)
- [DISCO: Distributed Inference with Sparse Communications](https://mlsanity.com/paper/2302.11180/) (2023)
- [RockNet: Distributed Learning on Ultra-Low-Power Devices](https://mlsanity.com/paper/2510.13320/) (2025)

## Abstract (arXiv)

Transformer models are rapidly becoming a cornerstone of modern Internet of Things (IoT) applications, yet their computational and memory demands far exceed the capabilities of a single typical ultra-low-power IoT device. We present CATS, a framework for distributed transformer inference on ultra-low-power wireless devices, enabling multiple devices to collaboratively execute models far larger than what a single device can sustain. At its core, CATS is a communication-aware distributed transformer inference scheme co-designed across transformer partitioning, wireless communication and training. It employs SomeGather, a new pruned communication primitive that selectively broadcasts activation columns to reduce communication bandwidth and RAM usage without sacrificing model accuracy. Building on SomeGather, we design a partitioning method that exploits this primitive for efficient model parallelism. To cope with unreliable wireless communication, CATS employs message-dropout during training, which mimics packet losses and yields models that are robust to message loss during inference. In real-world experiments, we show that CATS brings distributed transformer inference to ultra-low-power wireless devices for the first time, with deployments on up to 16 devices that collaboratively execute transformer models up to 14 times larger than what a single device can run.