mlsanityData to 7 Oct 2026

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

arXiv 2103.06695 ↗First posted 11 Mar 2021eess.AS cs.LG cs.SDcited by 106

Not on the map. Why?

Abstract (arXiv)

Inspired by the recent progress in self-supervised learning for computer vision that generates supervision using data augmentations, we explore a new general-purpose audio representation learning approach. We propose learning general-purpose audio representation from a single audio segment without expecting relationships between different time segments of audio samples. To implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. Unlike most previous audio self-supervised learning methods that rely on agreement of vicinity audio segments or disagreement of remote ones, BYOL-A creates contrasts in an augmented audio segment pair derived from a single audio segment. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive ablation studies also clarified the contribution of each component and their combinations.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

29 of 35 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
17 more
  1. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  2. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  3. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  4. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  5. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  6. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  7. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  8. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  9. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  10. 2020Towards Domain-Agnostic Contrastive Learningcited by 75
  11. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  12. 2020Unsupervised Contrastive Learning of Sound Event Representationscited by 32
  13. 2020i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learningcited by 19
  14. 2018Deep Convolutional Neural Network with Mixup for Environmental Sound Classificationcited by 15
  15. 2020COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representationscited by 13
  16. 2020Effects of Word-frequency based Pre- and Post- Processings for Audio Captioningcited by 12
  17. 2020The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 31; 2023: 20; 2024: 21; 2025: 16; 2026: 10.

106 papers.

Sort:

By area:

  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  3. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  4. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  5. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  6. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  7. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  8. 2022Audio Self-supervised Learning: A Surveycited by 65
  9. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  10. 2024What to align in multimodal contrastive learning?cited by 41
  11. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  12. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
38 more
  1. 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
  2. 2022Contrastive Audio-Language Learning for Musiccited by 32
  3. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  4. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  5. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  6. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  7. 2021SERAB: A multi-lingual benchmark for speech emotion recognitioncited by 22
  8. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  9. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  10. 2024Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulationcited by 14
  11. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  12. 2022ATST: Audio Representation Learning with Teacher-Student Transformercited by 13
  13. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13
  14. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  15. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  16. 2022That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulationcited by 12
  17. 2021One Billion Audio Sounds from GPU-enabled Modular Synthesiscited by 11
  18. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  19. 2022Contrastive Learning with Positive-Negative Frame Mask for Music Representationcited by 10
  20. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 9
  21. 2022Equivariant Self-Supervision for Musical Tempo Estimationcited by 9
  22. 2023EnCodecMAE: Leveraging neural codecs for universal audio representation learningcited by 9
  23. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  24. 2022Self-supervised Graphs for Audio Representation Learning with Limited Labeled Datacited by 8
  25. 2022Learning Representations for New Sound Classes With Continual Self-Supervised Learningcited by 8
  26. 2022SampleMatch: Drum Sample Retrieval by Musical Contextcited by 8
  27. 2023Self-Supervised Learning for Anomalous Sound Detectioncited by 8
  28. 2025A Multimodal Deviation Perceiving Framework for Weakly-Supervised Temporal Forgery Localizationcited by 8
  29. 2021DECAR: Deep Clustering for learning general-purpose Audio Representationscited by 7
  30. 2021Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicitycited by 7
  31. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  32. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  33. 2022Similarity learning for wells based on logging datacited by 6
  34. 2022Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Loadcited by 6
  35. 2022Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learningcited by 6
  36. 2024ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalitiescited by 6
  37. 2022AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Workcited by 5
  38. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settingscited by 5

50 most cited of 106 All 106 (JSON)