mlsanityData to 14 Oct 2026

MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers

arXiv 2410.17957 ↗First posted 23 Oct 2024cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose MCUBERT to enable language models like BERT on tiny microcontroller units (MCUs) through network and scheduling co-optimization. We observe the embedding table contributes to the major storage bottleneck for tiny BERT models. Hence, at the network level, we propose an MCU-aware two-stage neural architecture search algorithm based on clustered low-rank approximation for embedding compression. To reduce the inference memory requirements, we further propose a novel fine-grained MCU-friendly scheduling strategy. Through careful computation tiling and re-ordering as well as kernel design, we drastically increase the input sequence lengths supported on MCUs without any latency or accuracy penalty. MCUBERT reduces the parameter size of BERT-tiny and BERT-mini by 5.7× and 3.0× and the execution memory by 3.5× and 4.3×, respectively. MCUBERT also achieves 1.5× latency reduction. For the first time, MCUBERT enables lightweight BERT models on commodity MCUs and processing more than 512 tokens with less than 256KB of memory.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

30 of 46 references.

References · 30

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  6. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  7. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
  8. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  9. 2022Language model compression with weighted low-rank factorizationcited by 195
  10. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  11. 2020TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systemscited by 165
  12. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
18 more
  1. 2018CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUscited by 121
  2. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  3. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  4. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
  5. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  6. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  7. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  8. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  9. 2019SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollerscited by 52
  10. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  11. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  12. 2019Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollerscited by 33
  13. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  14. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  15. 2022UDC: Unified DNAS for Compressible TinyML Modelscited by 10
  16. 2023MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memorycited by 9
  17. 2022AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Modelscited by 7
  18. 2019Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decompositioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  2. 2024Transformer-Based Fault-Tolerant Control for Fixed-Wing UAVs Using Knowledge Distillation and In-Context Adaptationcited by 3
  3. 2026Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysiscited by 0