mlsanityData to 7 Oct 2026

Layer-wise Analysis of a Self-supervised Speech Representation Model

arXiv 2107.04734 ↗First posted 10 Jul 2021cs.CL cs.LG eess.AScited by 293

Not on the map. Why?

Abstract (arXiv)

Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.

Key facts. Cited by 293 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 10 Jul 2021.

Builds on

37 of 53 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
25 more
  1. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  3. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  4. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  5. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  6. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  7. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  8. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  9. 2021Unsupervised Speech Recognitioncited by 147
  10. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  11. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  12. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  13. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  14. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  15. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  16. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  17. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38
  18. 2017Representations of language in a model of visually grounded speech signalcited by 37
  19. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  20. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  21. 2020Probing Acoustic Representations for Phonetic Propertiescited by 26
  22. 2020Evaluating the reliability of acoustic speech embeddingscited by 21
  23. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  24. 2020Multilingual Jointly Trained Acoustic and Written Word Embeddingscited by 16
  25. 2019Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognitioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 46; 2023: 54; 2024: 61; 2025: 45; 2026: 80.

293 papers.

Sort:

By area:

  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  6. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  7. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  8. 2023Voice Conversion With Just Nearest Neighborscited by 67
  9. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  10. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  11. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  12. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
38 more
  1. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  2. 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
  3. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 35
  4. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  5. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  6. 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 30
  7. 2022Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learningcited by 29
  8. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  9. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  10. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  11. 2022Pseudo Label Is Better Than Human Labelcited by 25
  12. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  13. 2024Encoding of lexical tone in self-supervised models of spoken languagecited by 22
  14. 2021Fusing ASR Outputs in Joint Training for Speech Emotion Recognitioncited by 21
  15. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  16. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  17. 2024TransformerFAM: Feedback attention is working memorycited by 21
  18. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 19
  19. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  20. 2022Opening the Black Box of wav2vec Feature Encodercited by 18
  21. 2023Self-supervised representations in speech-based depression detectioncited by 18
  22. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  23. 2023What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Modelcited by 18
  24. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  25. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  26. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  27. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  28. 2024Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detectioncited by 17
  29. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  30. 2023Self-Supervised Models of Speech Infer Universal Articulatory Kinematicscited by 16
  31. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  32. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  33. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  34. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  35. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  36. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  37. 2022A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systemscited by 14
  38. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14

50 most cited of 293 All 293 (JSON)