mlsanityData to 7 Oct 2026

Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

arXiv 2210.02595 ↗First posted 5 Oct 2022eess.AS cs.CL cs.SDcited by 31

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

16 of 41 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  8. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  9. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  10. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  11. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  12. 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 47
4 more
  1. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  2. 2021Fusing ASR Outputs in Joint Training for Speech Emotion Recognitioncited by 21
  3. 2022SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speechcited by 20
  4. 2020Recognizing More Emotions with Less Data Using Self-supervised Transfer Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 15; 2025: 1; 2026: 4.

31 papers.

Sort:

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  2. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  3. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  4. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  5. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  6. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  7. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  8. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
  9. 2023ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognitioncited by 7
  10. 2024A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognitioncited by 6
  11. 2024Anonymising Elderly and Pathological Speech: Voice Conversion Using DDSP and Query-by-Examplecited by 5
  12. 2023On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesiscited by 4
19 more
  1. 2023Comparative Analysis of the wav2vec 2.0 Feature Extractorcited by 4
  2. 2023Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognitioncited by 4
  3. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  4. 2023A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTScited by 3
  5. 2024Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluationcited by 3
  6. 2024Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Modelscited by 3
  7. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  8. 2024Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Featurescited by 2
  9. 2024Disentangling Textual and Acoustic Features of Neural Speech Representationscited by 2
  10. 2024Speech foundation models in healthcare: Effect of layer selection on pathological speech feature predictioncited by 1
  11. 2024Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual modelscited by 0
  12. 2024Beyond the Labels: Unveiling Text-Dependency in Paralinguistic Speech Recognition Datasetscited by 0
  13. 2024Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotationscited by 0
  14. 2024Addressing Emotion Bias in Music Emotion Recognition and Generation with Frechet Audio Distancecited by 0
  15. 2024Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representationscited by 0
  16. 2026Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Researchcited by 0
  17. 2026Two-Stage Multimodal Framework for Emotion Mimicry Intensity Predictioncited by 0
  18. 2026Iterative Self-Learning for Expressive Text-to-Speech Synthesiscited by 0
  19. 2026Why Alzheimer's Speech Screening Fails to Generalize: Bridging the Deployment Gap via Cross-Corpus Evidence Anchoringcited by 0