mlsanityData to 14 Oct 2026

Opening the Black Box of wav2vec Feature Encoder

arXiv 2210.15386 ↗First posted 27 Oct 2022cs.SD cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Self-supervised models, namely, wav2vec and its variants, have shown promising results in various downstream tasks in the speech domain. However, their inner workings are poorly understood, calling for in-depth analyses on what the model learns. In this paper, we concentrate on the convolutional feature encoder where its latent space is often speculated to represent discrete acoustic units. To analyze the embedding space in a reductive manner, we feed the synthesized audio signals, which is the summation of simple sine waves. Through extensive experiments, we conclude that various information is embedded inside the feature encoder representations: (1) fundamental frequency, (2) formants, and (3) amplitude, packed with (4) sufficient temporal detail. Further, the information incorporated inside the latent representations is analogous to spectrograms but with a fundamental difference: latent representations construct a metric space so that closer representations imply acoustic similarity.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 13 of its 23 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

13 of 23 references.

References · 13

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  7. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  8. 2016Learning Features of Music from Scratchcited by 58
  9. 2021Towards Learning Universal Audio Representationscited by 51
  10. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 33
  11. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  12. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 15
1 more
  1. 2013Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 6; 2025: 4; 2026: 4.

18 papers.

Sort:

  1. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  2. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  3. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  4. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  5. 2026[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmeticcited by 12
  6. 2023Speech Intelligibility Assessment of Dysarthric Speech by using Goodness of Pronunciation with Uncertainty Quantificationcited by 10
  7. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  8. 2024Self-supervised Speech Representations Still Struggle with African American Vernacular Englishcited by 6
  9. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  10. 2023Comparative Analysis of the wav2vec 2.0 Feature Extractorcited by 4
  11. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  12. 2024A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speechcited by 2
6 more
  1. 2025Automatic classification of stop realisation with wav2vec2.0cited by 1
  2. 2023Unsupervised Speech Representation Pooling Using Vector Quantizationcited by 0
  3. 2024Transliterated Zero-Shot Domain Adaptation for Automatic Speech Recognitioncited by 0
  4. 2024Voice Biomarker Analysis and Automated Severity Classification of Dysarthric Speech in a Multilingual Contextcited by 0
  5. 2026Beyond task performance: Decoding bioacoustic embeddings with speech featurescited by 0
  6. 2026Phone Segmentation and Recognition through Phonological Activation Mappingcited by 0