# MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory

arXiv: https://arxiv.org/abs/2310.16898 · HTML: https://mlsanity.com/paper/2310.16898/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [Embodied Task Planning with Large Language Models](https://mlsanity.com/paper/2307.01848/) (2023)
- [SeaDronesSee: A Maritime Benchmark for Detecting Humans in Open Water](https://mlsanity.com/paper/2105.01922/) (2021)
- [GLiT: Neural Architecture Search for Global and Local Image Transformer](https://mlsanity.com/paper/2107.02960/) (2021)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)

## Built on by (most cited first, 9 of 9)

- [ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models](https://mlsanity.com/paper/2412.00447/) (2024)
- [TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors](https://mlsanity.com/paper/2407.21453/) (2024)
- [MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers](https://mlsanity.com/paper/2410.17957/) (2024)
- [Transformer-Based Fault-Tolerant Control for Fixed-Wing UAVs Using Knowledge Distillation and In-Context Adaptation](https://mlsanity.com/paper/2411.02975/) (2024)
- [ELASTIC: Efficient Once For All Iterative Search for Object Detection on Microcontrollers](https://mlsanity.com/paper/2503.21999/) (2025)
- [TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices](https://mlsanity.com/paper/2311.01759/) (2023)
- [Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutions](https://mlsanity.com/paper/2508.08352/) (2025)
- [Towards Real-Time ECG and EMG Modeling on μNPUs](https://mlsanity.com/paper/2604.18067/) (2026)
- [FTerViT: Fully Ternary Vision Transformer](https://mlsanity.com/paper/2605.21171/) (2026)

## Abstract (arXiv)

Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neural network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at https://github.com/liangyn22/MCUFormer.