# DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers

arXiv: https://arxiv.org/abs/2403.09035 · HTML: https://mlsanity.com/paper/2403.09035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems](https://mlsanity.com/paper/2010.08678/) (2020)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones](https://mlsanity.com/paper/1611.07688/) (2016)
- [Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances](https://mlsanity.com/paper/2111.00418/) (2021)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks](https://mlsanity.com/paper/2110.05668/) (2021)
- [Real-time Short Video Recommendation on Mobile Devices](https://mlsanity.com/paper/2208.09577/) (2022)
- [Differentiable Network Pruning for Microcontrollers](https://mlsanity.com/paper/2110.08350/) (2021)
- [Neural networks on microcontrollers: saving memory at inference via operator reordering](https://mlsanity.com/paper/1910.05110/) (2019)
- [TinyML for Ubiquitous Edge AI](https://mlsanity.com/paper/2102.01255/) (2021)
- [Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms](https://mlsanity.com/paper/2001.10876/) (2020)
- [Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks](https://mlsanity.com/paper/2209.00839/) (2022)
- [Pex: Memory-efficient Microcontroller Deep Learning through Partial Execution](https://mlsanity.com/paper/2211.17246/) (2022)

## Abstract (arXiv)

Enabling efficient and accurate deep neural network (DNN) inference on microcontrollers is non-trivial due to the constrained on-chip resources. Current methodologies primarily focus on compressing larger models yet at the expense of model accuracy. In this paper, we rethink the problem from the inverse perspective by constructing small/weak models directly and improving their accuracy. Thus, we introduce DiTMoS, a novel DNN training and inference framework with a selector-classifiers architecture, where the selector routes each input sample to the appropriate classifier for classification. DiTMoS is grounded on a key insight: a composition of weak models can exhibit high diversity and the union of them can significantly boost the accuracy upper bound. To approach the upper bound, DiTMoS introduces three strategies including diverse training data splitting to increase the classifiers' diversity, adversarial selector-classifiers training to ensure synergistic interactions thereby maximizing their complementarity, and heterogeneous feature aggregation to improve the capacity of classifiers. We further propose a network slicing technique to alleviate the extra memory overhead incurred by feature aggregation. We deploy DiTMoS on the Neucleo STM32F767ZI board and evaluate it based on three time-series datasets for human activity recognition, keywords spotting, and emotion recognition, respectively. The experiment results manifest that: (a) DiTMoS achieves up to 13.4% accuracy improvement compared to the best baseline; (b) network slicing almost completely eliminates the memory overhead incurred by feature aggregation with a marginal increase of latency.