# DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data

arXiv: https://arxiv.org/abs/2104.11629 · HTML: https://mlsanity.com/paper/2104.11629/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network](https://mlsanity.com/paper/1811.11431/) (2018)
- [SqueezeNext: Hardware-Aware Neural Network Design](https://mlsanity.com/paper/1803.10615/) (2018)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [Accelerate CNN via Recursive Bayesian Pruning](https://mlsanity.com/paper/1812.00353/) (2018)
- [EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2103.08310/) (2021)
- [auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1712.04382/) (2017)
- [Overcoming data scarcity with transfer learning](https://mlsanity.com/paper/1711.05099/) (2017)
- [End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning](https://mlsanity.com/paper/1802.01115/) (2018)
- [The voice of COVID-19: Acoustic correlates of infection](https://mlsanity.com/paper/2012.09478/) (2020)
- [AReLU: Attention-based Rectified Linear Unit](https://mlsanity.com/paper/2006.13858/) (2020)
- [SapAugment: Learning A Sample Adaptive Policy for Data Augmentation](https://mlsanity.com/paper/2011.01156/) (2020)

## Built on by (most cited first, 12 of 12)

- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [SpliceOut: A Simple and Efficient Audio Augmentation Method](https://mlsanity.com/paper/2110.00046/) (2021)
- [The ACM Multimedia 2022 Computational Paralinguistics Challenge: Vocalisations, Stuttering, Activity, & Mosquitoes](https://mlsanity.com/paper/2205.06799/) (2022)
- [Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results](https://mlsanity.com/paper/2209.14272/) (2022)
- [The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition](https://mlsanity.com/paper/2406.07753/) (2024)
- [METEOR Guided Divergence for Video Captioning](https://mlsanity.com/paper/2212.10690/) (2022)
- [The Broad Impact of Feature Imitation: Neural Enhancements Across Financial, Speech, and Physiological Domains](https://mlsanity.com/paper/2309.12279/) (2023)
- [autrainer: A Modular and Extensible Deep Learning Toolkit for Computer Audition Tasks](https://mlsanity.com/paper/2412.11943/) (2024)
- [Neurosymbolic Artificial Intelligence for Robust Network Intrusion Detection: From Scratch to Transfer Learning](https://mlsanity.com/paper/2506.04454/) (2025)
- [Climate Change & Computer Audition: A Call to Action and Overview on Audio Intelligence to Help Save the Planet](https://mlsanity.com/paper/2203.06064/) (2022)
- [Neurosymbolic AI Transfer Learning Improves Network Intrusion Detection](https://mlsanity.com/paper/2509.10850/) (2025)

## Abstract (arXiv)

Deep neural speech and audio processing systems have a large number of trainable parameters, a relatively complex architecture, and require a vast amount of training data and computational power. These constraints make it more challenging to integrate such systems into embedded devices and utilise them for real-time, real-world applications. We tackle these limitations by introducing DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition using pre-trained image convolutional neural networks (CNNs). The framework creates and augments Mel-spectrogram plots on-the-fly from raw audio signals which are then used to finetune specific pre-trained CNNs for the target classification task. Subsequently, the whole pipeline can be run in real-time with a mean inference lag of 242.0 ms when a DenseNet121 model is used on a consumer-grade Motorola moto e7 plus smartphone. DeepSpectrumLite operates decentralised, eliminating the need for data upload for further processing. By obtaining state-of-the-art results on a set of paralinguistics tasks, we demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing, even when data is scarce. We provide an extensive command-line interface for users and developers which is comprehensively documented and publicly available at https://github.com/DeepSpectrum/DeepSpectrumLite.