# Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting

arXiv: https://arxiv.org/abs/2309.08108 · HTML: https://mlsanity.com/paper/2309.08108/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2023.

## Built on by (most cited first, 17 of 17)

- [Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques](https://mlsanity.com/paper/2406.08353/) (2024)
- [From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs](https://mlsanity.com/paper/2408.17026/) (2024)
- [Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition](https://mlsanity.com/paper/2409.09785/) (2024)
- [AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models](https://mlsanity.com/paper/2409.18339/) (2024)
- [Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correction](https://mlsanity.com/paper/2409.15551/) (2024)
- [Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts](https://mlsanity.com/paper/2505.17222/) (2025)
- [A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition](https://mlsanity.com/paper/2311.07093/) (2023)
- [Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?](https://mlsanity.com/paper/2406.08800/) (2024)
- [GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text](https://mlsanity.com/paper/2407.11827/) (2024)
- [Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models](https://mlsanity.com/paper/2601.14620/) (2026)
- [Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models](https://mlsanity.com/paper/2410.03312/) (2024)
- [Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs](https://mlsanity.com/paper/2410.20334/) (2024)
- [Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction](https://mlsanity.com/paper/2506.10930/) (2025)
- [Speech Emotion Recognition Leveraging OpenAI's Whisper Representations and Attentive Pooling Methods](https://mlsanity.com/paper/2602.06000/) (2026)
- [TI-ASU: Toward Robust Automatic Speech Understanding through Text-to-speech Imputation Against Missing Speech Modality](https://mlsanity.com/paper/2404.17983/) (2024)
- [Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages](https://mlsanity.com/paper/2503.21806/) (2025)
- [How to Retrieve Examples in In-context Learning to Improve Conversational Emotion Recognition using Large Language Models?](https://mlsanity.com/paper/2506.20199/) (2025)

## Abstract (arXiv)

Significant advances are being made in speech emotion recognition (SER) using deep learning models. Nonetheless, training SER systems remains challenging, requiring both time and costly resources. Like many other machine learning tasks, acquiring datasets for SER requires substantial data annotation efforts, including transcription and labeling. These annotation processes present challenges when attempting to scale up conventional SER systems. Recent developments in foundational models have had a tremendous impact, giving rise to applications such as ChatGPT. These models have enhanced human-computer interactions including bringing unique possibilities for streamlining data collection in fields like SER. In this research, we explore the use of foundational models to assist in automating SER from transcription and annotation to augmentation. Our study demonstrates that these models can generate transcriptions to enhance the performance of SER systems that rely solely on speech data. Furthermore, we note that annotating emotions from transcribed speech remains a challenging task. However, combining outputs from multiple LLMs enhances the quality of annotations. Lastly, our findings suggest the feasibility of augmenting existing speech emotion datasets by annotating unlabeled speech samples.