# Pex: Memory-efficient Microcontroller Deep Learning through Partial Execution

arXiv: https://arxiv.org/abs/2211.17246 · HTML: https://mlsanity.com/paper/2211.17246/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems](https://mlsanity.com/paper/2010.08678/) (2020)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [Data Movement Is All You Need: A Case Study on Optimizing Transformers](https://mlsanity.com/paper/2007.00072/) (2020)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [μNAS: Constrained Neural Architecture Search for Microcontrollers](https://mlsanity.com/paper/2010.14246/) (2020)
- [SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers](https://mlsanity.com/paper/1905.12107/) (2019)
- [Visual Wake Words Dataset](https://mlsanity.com/paper/1906.05721/) (2019)
- [RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference](https://mlsanity.com/paper/2002.11921/) (2020)
- [DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device](https://mlsanity.com/paper/2104.09949/) (2021)
- [Differentiable Network Pruning for Microcontrollers](https://mlsanity.com/paper/2110.08350/) (2021)
- [Neural networks on microcontrollers: saving memory at inference via operator reordering](https://mlsanity.com/paper/1910.05110/) (2019)
- [Automap: Towards Ergonomic Automated Parallelism for ML Models](https://mlsanity.com/paper/2112.02958/) (2021)
- [UDC: Unified DNAS for Compressible TinyML Models](https://mlsanity.com/paper/2201.05842/) (2022)
- [Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism](https://mlsanity.com/paper/1903.06681/) (2019)
- [CoopNet: Cooperative Convolutional Neural Network for Low-Power MCUs](https://mlsanity.com/paper/1911.08606/) (2019)
- [Memory Safe Computations with XLA Compiler](https://mlsanity.com/paper/2206.14148/) (2022)

## Built on by (most cited first, 3 of 3)

- [MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design Framework for MCUs](https://mlsanity.com/paper/2407.18267/) (2024)
- [DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers](https://mlsanity.com/paper/2403.09035/) (2024)
- [TinySeg: Model Optimizing Framework for Image Segmentation on Tiny Embedded Systems](https://mlsanity.com/paper/2405.01857/) (2024)

## Abstract (arXiv)

Embedded and IoT devices, largely powered by microcontroller units (MCUs), could be made more intelligent by leveraging on-device deep learning. One of the main challenges of neural network inference on an MCU is the extremely limited amount of read-write on-chip memory (SRAM, < 512 kB). SRAM is consumed by the neural network layer (operator) input and output buffers, which, traditionally, must be in memory (materialised) for an operator to execute. We discuss a novel execution paradigm for microcontroller deep learning, which modifies the execution of neural networks to avoid materialising full buffers in memory, drastically reducing SRAM usage with no computation overhead. This is achieved by exploiting the properties of operators, which can consume/produce a fraction of their input/output at a time. We describe a partial execution compiler, Pex, which produces memory-efficient execution schedules automatically by identifying subgraphs of operators whose execution can be split along the feature ("channel") dimension. Memory usage is reduced further by targeting memory bottlenecks with structured pruning, leading to the co-design of the network architecture and its execution schedule. Our evaluation of image and audio classification models: (a) establishes state-of-the-art performance in low SRAM usage regimes for considered tasks with up to +2.9% accuracy increase; (b) finds that a 4x memory reduction is possible by applying partial execution alone, or up to 10.5x when using the compiler-pruning co-design, while maintaining the classification accuracy compared to prior work; (c) uses the recovered SRAM to process higher resolution inputs instead, increasing accuracy by up to +3.9% on Visual Wake Words.