# Self-supervised audio representation learning for mobile devices

arXiv: https://arxiv.org/abs/1905.11796 · HTML: https://mlsanity.com/paper/1905.11796/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 17 of 17)

- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Learning strides in convolutional neural networks](https://mlsanity.com/paper/2202.01653/) (2022)
- [Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling](https://mlsanity.com/paper/2104.04323/) (2021)
- [Labeling Chaos to Learning Harmony: Federated Learning with Noisy Labels](https://mlsanity.com/paper/2208.09378/) (2022)
- [DECAR: Deep Clustering for learning general-purpose Audio Representations](https://mlsanity.com/paper/2110.08895/) (2021)
- [The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image](https://mlsanity.com/paper/2112.00725/) (2021)
- [Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning](https://mlsanity.com/paper/2202.02611/) (2022)
- [Self-Supervised PPG Representation Learning Shows High Inter-Subject Variability](https://mlsanity.com/paper/2212.04902/) (2022)
- [Federated Self-Training for Semi-Supervised Audio Recognition](https://mlsanity.com/paper/2107.06877/) (2021)
- [Sound and Visual Representation Learning with Multiple Pretraining Tasks](https://mlsanity.com/paper/2201.01046/) (2022)
- [Federated Self-Supervised Learning for Acoustic Event Classification](https://mlsanity.com/paper/2203.11997/) (2022)
- [Coherent, super resolved radar beamforming using self-supervised learning](https://mlsanity.com/paper/2106.13085/) (2021)
- [Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks](https://mlsanity.com/paper/2104.03899/) (2021)
- [Self-Supervised Beat Tracking in Musical Signals with Polyphonic Contrastive Learning](https://mlsanity.com/paper/2201.01771/) (2022)
- [Homophone Reveals the Truth: A Reality Check for Speech2Vec](https://mlsanity.com/paper/2209.10791/) (2022)
- [Non-Contrastive Learning-based Behavioural Biometrics for Smart IoT Devices](https://mlsanity.com/paper/2210.12964/) (2022)

## Abstract (arXiv)

We explore self-supervised models that can be potentially deployed on mobile devices to learn general purpose audio representations. Specifically, we propose methods that exploit the temporal context in the spectrogram domain. One method estimates the temporal gap between two short audio segments extracted at random from the same audio clip. The other methods are inspired by Word2Vec, a popular technique used to learn word embeddings, and aim at reconstructing a temporal spectrogram slice from past and future slices or, alternatively, at reconstructing the context of surrounding slices from the current slice. We focus our evaluation on small encoder architectures, which can be potentially run on mobile devices during both inference (re-using a common learned representation across multiple downstream tasks) and training (capturing the true data distribution without compromising users' privacy when combined with federated learning). We evaluate the quality of the embeddings produced by the self-supervised learning models, and show that they can be re-used for a variety of downstream tasks, and for some tasks even approach the performance of fully supervised models of similar size.