# BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

arXiv: https://arxiv.org/abs/2103.06695 · HTML: https://mlsanity.com/paper/2103.06695/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Towards Domain-Agnostic Contrastive Learning](https://mlsanity.com/paper/2011.04419/) (2020)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Unsupervised Contrastive Learning of Sound Event Representations](https://mlsanity.com/paper/2011.07616/) (2020)
- [i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning](https://mlsanity.com/paper/2010.08887/) (2020)
- [Deep Convolutional Neural Network with Mixup for Environmental Sound Classification](https://mlsanity.com/paper/1808.08405/) (2018)
- [COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations](https://mlsanity.com/paper/2006.08386/) (2020)
- [Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning](https://mlsanity.com/paper/2009.11436/) (2020)
- [The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation](https://mlsanity.com/paper/2007.00225/) (2020)

## Built on by (most cited first, 30 of 106)

- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset](https://mlsanity.com/paper/2311.15308/) (2023)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning](https://mlsanity.com/paper/2205.06226/) (2022)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation](https://mlsanity.com/paper/2310.16917/) (2023)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks](https://mlsanity.com/paper/2306.04186/) (2023)
- [SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis](https://mlsanity.com/paper/2310.15247/) (2023)
- [Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking](https://mlsanity.com/paper/2406.16148/) (2024)
- [SERAB: A multi-lingual benchmark for speech emotion recognition](https://mlsanity.com/paper/2110.03414/) (2021)
- [AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models](https://mlsanity.com/paper/2309.10787/) (2023)
- [SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes](https://mlsanity.com/paper/2506.12222/) (2025)
- [Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation](https://mlsanity.com/paper/2405.08576/) (2024)
- [animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics](https://mlsanity.com/paper/2406.01253/) (2024)
- [T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data](https://mlsanity.com/paper/2410.05016/) (2024)
- [ATST: Audio Representation Learning with Teacher-Student Transformer](https://mlsanity.com/paper/2204.12076/) (2022)
- [ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification](https://mlsanity.com/paper/2211.13189/) (2022)
- [SSDM: Scalable Speech Dysfluency Modeling](https://mlsanity.com/paper/2408.16221/) (2024)
- [That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation](https://mlsanity.com/paper/2210.01116/) (2022)
- [One Billion Audio Sounds from GPU-enabled Modular Synthesis](https://mlsanity.com/paper/2104.12922/) (2021)
- [Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners](https://mlsanity.com/paper/2306.00561/) (2023)

## Abstract (arXiv)

Inspired by the recent progress in self-supervised learning for computer vision that generates supervision using data augmentations, we explore a new general-purpose audio representation learning approach. We propose learning general-purpose audio representation from a single audio segment without expecting relationships between different time segments of audio samples. To implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. Unlike most previous audio self-supervised learning methods that rely on agreement of vicinity audio segments or disagreement of remote ones, BYOL-A creates contrasts in an augmented audio segment pair derived from a single audio segment. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive ablation studies also clarified the contribution of each component and their combinations.