# Audio Self-supervised Learning: A Survey

arXiv: https://arxiv.org/abs/2203.01205 · HTML: https://mlsanity.com/paper/2203.01205/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 148 of its 197 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 30 of 148)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)

## Built on by (most cited first, 30 of 65)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning](https://mlsanity.com/paper/2305.10005/) (2023)
- [Next-Latent Prediction Transformers Learn Compact World Models](https://mlsanity.com/paper/2511.05963/) (2025)
- [Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks](https://mlsanity.com/paper/2306.04186/) (2023)
- [N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://mlsanity.com/paper/2306.02902/) (2023)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [Probing Speech Emotion Recognition Transformers for Linguistic Knowledge](https://mlsanity.com/paper/2204.00400/) (2022)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [Investigating Pre-trained Audio Encoders in the Low-Resource Condition](https://mlsanity.com/paper/2305.17733/) (2023)
- [Self-supervised visual learning in the low-data regime: a comparative evaluation](https://mlsanity.com/paper/2404.17202/) (2024)
- [BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding](https://mlsanity.com/paper/2408.06851/) (2024)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition](https://mlsanity.com/paper/2310.18877/) (2023)
- [A vector quantized masked autoencoder for speech emotion recognition](https://mlsanity.com/paper/2304.11117/) (2023)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [Self-Supervised Learning for Anomalous Sound Detection](https://mlsanity.com/paper/2312.09578/) (2023)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [A vector quantized masked autoencoder for audiovisual speech emotion recognition](https://mlsanity.com/paper/2305.03568/) (2023)
- [R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces](https://mlsanity.com/paper/2311.09117/) (2023)
- [MTI-Net: A Multi-Target Speech Intelligibility Prediction Model](https://mlsanity.com/paper/2204.03310/) (2022)
- [Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation](https://mlsanity.com/paper/2305.11685/) (2023)
- [MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning](https://mlsanity.com/paper/2508.12709/) (2025)
- [Audio Barlow Twins: Self-Supervised Audio Representation Learning](https://mlsanity.com/paper/2209.14345/) (2022)
- [Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesis](https://mlsanity.com/paper/2504.13386/) (2025)
- [ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development](https://mlsanity.com/paper/2307.08720/) (2023)
- [A Review on Self-Supervised Learning for Time Series Anomaly Detection: Recent Advances and Open Challenges](https://mlsanity.com/paper/2501.15196/) (2025)

## Abstract (arXiv)

Inspired by the humans' cognitive ability to generalise knowledge and skills, Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations, which is an expensive and time consuming task. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing. Comprehensive reviews summarising the knowledge in audio SSL are currently missing. To fill this gap, in the present work, we provide an overview of the SSL methods used for audio and speech processing applications. Herein, we also summarise the empirical works that exploit the audio modality in multi-modal SSL frameworks, and the existing suitable benchmarks to evaluate the power of SSL in the computer audition domain. Finally, we discuss some open problems and point out the future directions on the development of audio SSL.