mlsanityData to 7 Oct 2026

Audio Self-supervised Learning: A Survey

arXiv 2203.01205 ↗First posted 2 Mar 2022cs.SD cs.AI eess.AScited by 65

Not on the map. Why?

Abstract (arXiv)

Inspired by the humans' cognitive ability to generalise knowledge and skills, Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations, which is an expensive and time consuming task. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing. Comprehensive reviews summarising the knowledge in audio SSL are currently missing. To fill this gap, in the present work, we provide an overview of the SSL methods used for audio and speech processing applications. Herein, we also summarise the empirical works that exploit the audio modality in multi-modal SSL frameworks, and the existing suitable benchmarks to evaluate the power of SSL in the computer audition domain. Finally, we discuss some open problems and point out the future directions on the development of audio SSL.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 148 of its 197 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

148 of 197 references.

References · 148

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  14. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  15. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  16. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  17. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  18. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  19. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  20. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  21. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  22. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  23. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  24. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  25. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  26. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  27. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  28. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  29. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  30. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  31. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  32. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  33. 2017Large Batch Training of Convolutional Networkscited by 529
  34. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  35. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  36. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  37. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  38. 2017Look, Listen and Learncited by 452

50 most cited of 148 All 148 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 28; 2024: 10; 2025: 13; 2026: 5.

65 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  4. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  5. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  6. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  7. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  8. 2023N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognitioncited by 20
  9. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  10. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  11. 2022Probing Speech Emotion Recognition Transformers for Linguistic Knowledgecited by 13
  12. 2023The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisationcited by 12
38 more
  1. 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
  2. 2024Self-supervised visual learning in the low-data regime: a comparative evaluationcited by 11
  3. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  4. 2023Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognitioncited by 9
  5. 2024BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embeddingcited by 9
  6. 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
  7. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  8. 2023Self-Supervised Learning for Anomalous Sound Detectioncited by 8
  9. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  10. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  11. 2022MTI-Net: A Multi-Target Speech Intelligibility Prediction Modelcited by 6
  12. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 6
  13. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  14. 2025Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesiscited by 5
  15. 2025MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learningcited by 5
  16. 2022Audio Barlow Twins: Self-Supervised Audio Representation Learningcited by 4
  17. 2023ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Developmentcited by 4
  18. 2025A Review on Self-Supervised Learning for Time Series Anomaly Detection: Recent Advances and Open Challengescited by 4
  19. 2025Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learningcited by 4
  20. 2023Unearthing InSights into Mars: Unsupervised Source Separation with Limited Datacited by 3
  21. 2023Multi-scale clustering and source separation of InSight mission seismic datacited by 3
  22. 2023MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representationscited by 3
  23. 2023Active Bird2Vec: Towards End-to-End Bird Sound Monitoring with Transformerscited by 3
  24. 2023A novel dual-stream time-frequency contrastive pretext tasks framework for sleep stage classificationcited by 3
  25. 2024The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognitioncited by 3
  26. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  27. 2022Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processingcited by 2
  28. 2023Phone and speaker spatial organization in self-supervised speech representationscited by 2
  29. 2023Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Taskscited by 2
  30. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  31. 2022Self-Relation Attention and Temporal Awareness for Emotion Recognition via Vocal Burstcited by 1
  32. 2022DDSupport: Language Learning Support System that Displays Differences and Distances from Model Speechcited by 1
  33. 2023Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representationscited by 1
  34. 2023Rethinking Samples Selection for Contrastive Learning: Mining of Potential Samplescited by 1
  35. 2024Spatial Context-based Self-Supervised Learning for Handwritten Text Recognitioncited by 1
  36. 2024Fine-grained Speech Sentiment Analysis in Chinese Psychological Support Hotlines Based on Large-scale Pre-trained Modelcited by 1
  37. 2024SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Modelscited by 1
  38. 2025Evaluation of Deep Audio Representations for Hearablescited by 1

50 most cited of 65 All 65 (JSON)