# Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

arXiv: https://arxiv.org/abs/2407.21315 · HTML: https://mlsanity.com/paper/2407.21315/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs](https://mlsanity.com/paper/2311.06753/) (2023)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion](https://mlsanity.com/paper/2404.17858/) (2024)
- [Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2404.17862/) (2024)
- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations](https://mlsanity.com/paper/2407.00119/) (2024)
- [When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection](https://mlsanity.com/paper/2402.13276/) (2024)
- [LaERC-S: Improving LLM-based Emotion Recognition in Conversation with Speaker Characteristics](https://mlsanity.com/paper/2403.07260/) (2024)
- [BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Tasks](https://mlsanity.com/paper/2407.04279/) (2024)
- [TEASEL: A Transformer-Based Speech-Prefixed Language Model](https://mlsanity.com/paper/2109.05522/) (2021)
- [Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions](https://mlsanity.com/paper/2406.14701/) (2024)

## Built on by (most cited first, 27 of 27)

- [Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects](https://mlsanity.com/paper/2505.20511/) (2025)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs](https://mlsanity.com/paper/2503.21480/) (2025)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues](https://mlsanity.com/paper/2502.10973/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models](https://mlsanity.com/paper/2508.07273/) (2025)
- [WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning](https://mlsanity.com/paper/2501.16344/) (2025)
- [VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation](https://mlsanity.com/paper/2602.06270/) (2026)
- [MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue](https://mlsanity.com/paper/2603.06194/) (2026)
- [Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts](https://mlsanity.com/paper/2503.06805/) (2025)
- [Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognition](https://mlsanity.com/paper/2506.02059/) (2025)
- [Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation](https://mlsanity.com/paper/2509.02278/) (2025)
- [Personality-guided Public-Private Domain Disentangled Hypergraph-Former Network for Multimodal Depression Detection](https://mlsanity.com/paper/2511.12460/) (2025)
- [Dynamic Stress Detection: A Study of Temporal Progression Modelling of Stress in Speech](https://mlsanity.com/paper/2510.08586/) (2025)
- [Empowering Dysarthric Speech: Leveraging Advanced LLMs for Accurate Speech Correction and Multimodal Emotion Analysis](https://mlsanity.com/paper/2410.12867/) (2024)
- [Exploring Audio Editing Features as User-Centric Privacy Defenses Against Large Language Model(LLM) Based Emotion Inference Attacks](https://mlsanity.com/paper/2501.18727/) (2025)
- [BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation](https://mlsanity.com/paper/2503.23990/) (2025)
- [How to Retrieve Examples in In-context Learning to Improve Conversational Emotion Recognition using Large Language Models?](https://mlsanity.com/paper/2506.20199/) (2025)
- [MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages](https://mlsanity.com/paper/2511.04914/) (2025)
- [SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2604.01916/) (2026)
- [VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2605.18547/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [DiaRelay: Relaying Dialogue Context with a Constant-Size Memory for Emotion Recognition in Conversation](https://mlsanity.com/paper/2608.22745/) (2026)
- [Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores](https://mlsanity.com/paper/2608.26137/) (2026)
- [Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding](https://mlsanity.com/paper/2609.32804/) (2026)
- [From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models](https://mlsanity.com/paper/2609.39453/) (2026)

## Abstract (arXiv)

Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiments demonstrate that incorporating speech descriptions yields a more than 2% increase in the average weighted F1 score on IEMOCAP (from 70.111% to 72.596%).