# Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment

arXiv: https://arxiv.org/abs/2407.17716 · HTML: https://mlsanity.com/paper/2407.17716/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [OpenStreetMap: Challenges and Opportunities in Machine Learning and Remote Sensing](https://mlsanity.com/paper/2007.06277/) (2020)
- [FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement](https://mlsanity.com/paper/2206.07293/) (2022)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation](https://mlsanity.com/paper/2203.16104/) (2022)
- [On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks](https://mlsanity.com/paper/1804.02173/) (2018)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [LanSER: Language-Model Supported Speech Emotion Recognition](https://mlsanity.com/paper/2309.03978/) (2023)
- [pMCT: Patched Multi-Condition Training for Robust Speech Recognition](https://mlsanity.com/paper/2207.04949/) (2022)

## Built on by (most cited first, 1 of 1)

- [Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition](https://mlsanity.com/paper/2509.08470/) (2025)

## Abstract (arXiv)

Speech emotion recognition (SER) systems often struggle in real-world environments, where ambient noise severely degrades their performance. This paper explores a novel approach that exploits prior knowledge of testing environments to maximize SER performance under noisy conditions. To address this task, we propose a text-guided, environment-aware training where an SER model is trained with contaminated speech samples and their paired noise description. We use a pre-trained text encoder to extract the text-based environment embedding and then fuse it to a transformer-based SER model during training and inference. We demonstrate the effectiveness of our approach through our experiment with the MSP-Podcast corpus and real-world additive noise samples collected from the Freesound and DEMAND repositories. Our experiment indicates that the text-based environment descriptions processed by a large language model (LLM) produce representations that improve the noise-robustness of the SER system. With a contrastive learning (CL)-based representation, our proposed method can be improved by jointly fine-tuning the text encoder with the emotion recognition model. Under the -5dB signal-to-noise ratio (SNR) level, fine-tuning the text encoder improves our CL-based representation method by 76.4% (arousal), 100.0% (dominance), and 27.7% (valence).