# A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2311.07093 · HTML: https://mlsanity.com/paper/2311.07093/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 81 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data](https://mlsanity.com/paper/2209.15329/) (2022)
- [Emotion recognition by fusing time synchronous and time asynchronous representations](https://mlsanity.com/paper/2010.14102/) (2020)
- [Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models](https://mlsanity.com/paper/1911.09762/) (2019)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting](https://mlsanity.com/paper/2309.08108/) (2023)
- [Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition](https://mlsanity.com/paper/2010.12733/) (2020)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [End-to-End Speech Emotion Recognition: Challenges of Real-Life Emergency Call Centers Data Recordings](https://mlsanity.com/paper/2110.14957/) (2021)

## Built on by (most cited first, 4 of 4)

- [MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction](https://mlsanity.com/paper/2401.13260/) (2024)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [Two-stage Framework for Robust Speech Emotion Recognition Using Target Speaker Extraction in Human Speech Noise Conditions](https://mlsanity.com/paper/2409.19585/) (2024)
- [Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2505.13983/) (2025)

## Abstract (arXiv)

This paper proposes an efficient attempt to noisy speech emotion recognition (NSER). Conventional NSER approaches have proven effective in mitigating the impact of artificial noise sources, such as white Gaussian noise, but are limited to non-stationary noises in real-world environments due to their complexity and uncertainty. To overcome this limitation, we introduce a new method for NSER by adopting the automatic speech recognition (ASR) model as a noise-robust feature extractor to eliminate non-vocal information in noisy speech. We first obtain intermediate layer information from the ASR model as a feature representation for emotional speech and then apply this representation for the downstream NSER task. Our experimental results show that 1) the proposed method achieves better NSER performance compared with the conventional noise reduction method, 2) outperforms self-supervised learning approaches, and 3) even outperforms text-based approaches using ASR transcription or the ground truth transcription of noisy speech.