# Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

arXiv: https://arxiv.org/abs/2210.02595 · HTML: https://mlsanity.com/paper/2210.02595/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech](https://mlsanity.com/paper/2203.03812/) (2022)
- [Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning](https://mlsanity.com/paper/2011.05585/) (2020)

## Built on by (most cited first, 30 of 31)

- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue](https://mlsanity.com/paper/2312.15316/) (2023)
- [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024)
- [Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition](https://mlsanity.com/paper/2309.10294/) (2023)
- [Crossmodal ASR Error Correction with Discrete Speech Units](https://mlsanity.com/paper/2405.16677/) (2024)
- [Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition](https://mlsanity.com/paper/2402.02617/) (2024)
- [ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition](https://mlsanity.com/paper/2305.16065/) (2023)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [Anonymising Elderly and Pathological Speech: Voice Conversion Using DDSP and Query-by-Example](https://mlsanity.com/paper/2410.15500/) (2024)
- [On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis](https://mlsanity.com/paper/2307.05132/) (2023)
- [Comparative Analysis of the wav2vec 2.0 Feature Extractor](https://mlsanity.com/paper/2308.04286/) (2023)
- [Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition](https://mlsanity.com/paper/2308.08713/) (2023)
- [Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation](https://mlsanity.com/paper/2409.10535/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS](https://mlsanity.com/paper/2303.02719/) (2023)
- [Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models](https://mlsanity.com/paper/2409.16920/) (2024)
- [On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillation](https://mlsanity.com/paper/2307.02720/) (2023)
- [Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Features](https://mlsanity.com/paper/2409.09511/) (2024)
- [Disentangling Textual and Acoustic Features of Neural Speech Representations](https://mlsanity.com/paper/2410.03037/) (2024)
- [Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction](https://mlsanity.com/paper/2402.01796/) (2024)
- [Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models](https://mlsanity.com/paper/2402.05581/) (2024)
- [Beyond the Labels: Unveiling Text-Dependency in Paralinguistic Speech Recognition Datasets](https://mlsanity.com/paper/2403.07767/) (2024)
- [Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotations](https://mlsanity.com/paper/2406.07900/) (2024)
- [Addressing Emotion Bias in Music Emotion Recognition and Generation with Frechet Audio Distance](https://mlsanity.com/paper/2409.15545/) (2024)
- [Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations](https://mlsanity.com/paper/2409.17899/) (2024)
- [Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research](https://mlsanity.com/paper/2604.25776/) (2026)
- [Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction](https://mlsanity.com/paper/2605.21869/) (2026)
- [Iterative Self-Learning for Expressive Text-to-Speech Synthesis](https://mlsanity.com/paper/2608.15910/) (2026)

## Abstract (arXiv)

Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.