# TinyM2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices

arXiv: https://arxiv.org/abs/2202.04303 · HTML: https://mlsanity.com/paper/2202.04303/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 17 of 17)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)
- [Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming](https://mlsanity.com/paper/2006.10518/) (2020)
- [Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data](https://mlsanity.com/paper/2006.05919/) (2020)
- [The COUGHVID crowdsourcing dataset: A corpus for the study of large-scale cough analysis algorithms](https://mlsanity.com/paper/2009.11644/) (2020)
- [Coswara -- A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosis](https://mlsanity.com/paper/2005.10548/) (2020)
- [Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments](https://mlsanity.com/paper/1707.05733/) (2017)
- [Ternary Neural Networks for Resource-Efficient AI Applications](https://mlsanity.com/paper/1609.00222/) (2016)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [Depthwise Separable Convolutions for Neural Machine Translation](https://mlsanity.com/paper/1706.03059/) (2017)
- [Structured Pruning of Neural Networks with Budget-Aware Regularization](https://mlsanity.com/paper/1811.09332/) (2018)
- [Cough Against COVID: Evidence of COVID-19 Signature in Cough Sounds](https://mlsanity.com/paper/2009.08790/) (2020)

## Built on by (most cited first, 2 of 2)

- [TinyVQA: Compact Multimodal Deep Neural Network for Visual Question Answering on Resource-Constrained Devices](https://mlsanity.com/paper/2404.03574/) (2024)
- [SeLoC-ML: Semantic Low-Code Engineering for Machine Learning Applications in Industrial IoT](https://mlsanity.com/paper/2207.08818/) (2022)

## Abstract (arXiv)

With the emergence of Artificial Intelligence (AI), new attention has been given to implement AI algorithms on resource constrained tiny devices to expand the application domain of IoT. Multimodal Learning has recently become very popular with the classification task due to its impressive performance for both image and audio event classification. This paper presents TinyM2Net -- a flexible system algorithm co-designed multimodal learning framework for resource constrained tiny devices. The framework was designed to be evaluated on two different case-studies: COVID-19 detection from multimodal audio recordings and battle field object detection from multimodal images and audios. In order to compress the model to implement on tiny devices, substantial network architecture optimization and mixed precision quantization were performed (mixed 8-bit and 4-bit). TinyM2Net shows that even a tiny multimodal learning model can improve the classification performance than that of any unimodal frameworks. The most compressed TinyM2Net achieves 88.4% COVID-19 detection accuracy (14.5% improvement from unimodal base model) and 96.8% battle field object detection accuracy (3.9% improvement from unimodal base model). Finally, we test our TinyM2Net models on a Raspberry Pi 4 to see how they perform when deployed to a resource constrained tiny device.