# Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices

arXiv: https://arxiv.org/abs/2207.05784 · HTML: https://mlsanity.com/paper/2207.05784/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition](https://mlsanity.com/paper/2205.03433/) (2022)
- [MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?](https://mlsanity.com/paper/2001.05936/) (2020)
- [Universal Paralinguistic Speech Representations Using Self-Supervised Conformers](https://mlsanity.com/paper/2110.04621/) (2021)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [Back to Simplicity: How to Train Accurate BNNs from Scratch?](https://mlsanity.com/paper/1906.08637/) (2019)
- [TRILLsson: Distilled Universal Paralinguistic Speech Representations](https://mlsanity.com/paper/2203.00236/) (2022)
- [FRILL: A Non-Semantic Speech Embedding for Mobile Devices](https://mlsanity.com/paper/2011.04609/) (2020)

## Abstract (arXiv)

This work introduces BRILLsson, a novel binary neural network-based representation learning model for a broad range of non-semantic speech tasks. We train the model with knowledge distillation from a large and real-valued TRILLsson model with only a fraction of the dataset used to train TRILLsson. The resulting BRILLsson models are only 2MB in size with a latency less than 8ms, making them suitable for deployment in low-resource devices such as wearables. We evaluate BRILLsson on eight benchmark tasks (including but not limited to spoken language identification, emotion recognition, health condition diagnosis, and keyword spotting), and demonstrate that our proposed ultra-light and low-latency models perform as well as large-scale models.