# FRILL: A Non-Semantic Speech Embedding for Mobile Devices

arXiv: https://arxiv.org/abs/2011.04609 · HTML: https://mlsanity.com/paper/2011.04609/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Universal Paralinguistic Speech Representations Using Self-Supervised Conformers](https://mlsanity.com/paper/2110.04621/) (2021)
- [TRILLsson: Distilled Universal Paralinguistic Speech Representations](https://mlsanity.com/paper/2203.00236/) (2022)
- [HeAR -- Health Acoustic Representations](https://mlsanity.com/paper/2403.02522/) (2024)
- [Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings](https://mlsanity.com/paper/2303.05737/) (2023)
- [Pretext Tasks selection for multitask self-supervised speech representation learning](https://mlsanity.com/paper/2107.00594/) (2021)
- [Label Aware Speech Representation Learning For Language Identification](https://mlsanity.com/paper/2306.04374/) (2023)
- [Health AI Developer Foundations](https://mlsanity.com/paper/2411.15128/) (2024)
- [Low-Energy On-Device Personalization for MCUs](https://mlsanity.com/paper/2403.08040/) (2024)
- [Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signals](https://mlsanity.com/paper/2309.05843/) (2023)
- [Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices](https://mlsanity.com/paper/2207.05784/) (2022)
- [MASR: Multi-label Aware Speech Representation](https://mlsanity.com/paper/2307.10982/) (2023)
- [Towards Characterizing Knowledge Distillation of PPG Heart Rate Estimation Models](https://mlsanity.com/paper/2511.18829/) (2025)

## Abstract (arXiv)

Learned speech representations can drastically improve performance on tasks with limited labeled data. However, due to their size and complexity, learned representations have limited utility in mobile settings where run-time performance can be a significant bottleneck. In this work, we propose a class of lightweight non-semantic speech embedding models that run efficiently on mobile devices based on the recently proposed TRILL speech embedding. We combine novel architectural modifications with existing speed-up techniques to create embedding models that are fast enough to run in real-time on a mobile device and exhibit minimal performance degradation on a benchmark of non-semantic speech tasks. One such model (FRILL) is 32x faster on a Pixel 1 smartphone and 40% the size of TRILL, with an average decrease in accuracy of only 2%. To our knowledge, FRILL is the highest-quality non-semantic embedding designed for use on mobile devices. Furthermore, we demonstrate that these representations are useful for mobile health tasks such as non-speech human sounds detection and face-masked speech detection. Our models and code are publicly available.