# LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations

arXiv: https://arxiv.org/abs/2501.11468 · HTML: https://mlsanity.com/paper/2501.11468/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2025.

## Built on by (most cited first, 9 of 9)

- [OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs](https://mlsanity.com/paper/2503.21480/) (2025)
- [GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.20919/) (2025)
- [When LLMs Team Up: The Emergence of Collaborative Affective Computing](https://mlsanity.com/paper/2506.01698/) (2025)
- [ABHINAYA -- A System for Speech Emotion Recognition In Naturalistic Conditions Challenge](https://mlsanity.com/paper/2505.18217/) (2025)
- [EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast](https://mlsanity.com/paper/2505.23732/) (2025)
- [Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound](https://mlsanity.com/paper/2511.12077/) (2025)
- [A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2602.23300/) (2026)
- [Dynamic Parameter Memory: Temporary LoRA-Enhanced LLM for Long-Sequence Emotion Recognition in Conversation](https://mlsanity.com/paper/2507.09076/) (2025)
- [Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition](https://mlsanity.com/paper/2609.20081/) (2026)

## Abstract (arXiv)

Emotion recognition in conversations (ERC) is challenging due to the multimodal nature of the emotion expression. In this paper, we propose to pretrain a text-based recognition model from unsupervised speech transcripts with LLM guidance. These transcriptions are obtained from a raw speech dataset with a pre-trained ASR system. A text LLM model is queried to provide pseudo-labels for these transcripts, and these pseudo-labeled transcripts are subsequently used for learning an utterance level text-based emotion recognition model. We use the utterance level text embeddings for emotion recognition in conversations along with speech embeddings obtained from a recently proposed pre-trained model. A hierarchical way of training the speech-text model is proposed, keeping in mind the conversational nature of the dataset. We perform experiments on three established datasets, namely, IEMOCAP, MELD, and CMU- MOSI, where we illustrate that the proposed model improves over other benchmarks and achieves state-of-the-art results on two out of these three datasets.