# InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models

arXiv: https://arxiv.org/abs/2309.11911 · HTML: https://mlsanity.com/paper/2309.11911/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks](https://mlsanity.com/paper/1708.04299/) (2017)
- [Directed Acyclic Graph Network for Conversational Emotion Recognition](https://mlsanity.com/paper/2105.12907/) (2021)
- [Making Language Models Better Tool Learners with Execution Feedback](https://mlsanity.com/paper/2305.13068/) (2023)
- [M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2206.02187/) (2022)
- [Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2210.08713/) (2022)
- [GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2207.12261/) (2022)
- [Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations](https://mlsanity.com/paper/2306.01505/) (2023)
- [Multi-Task Learning with Auxiliary Speaker Identification for Conversational Emotion Recognition](https://mlsanity.com/paper/2003.01478/) (2020)
- [EmotionIC: emotional inertia and contagion-driven dependency modeling for emotion recognition in conversation](https://mlsanity.com/paper/2303.11117/) (2023)
- [Hierarchical Dialogue Understanding with Special Tokens and Turn-level Attention](https://mlsanity.com/paper/2305.00262/) (2023)
- [Mimicking the Thinking Process for Emotion Recognition in Conversation with Prompts and Paraphrasing](https://mlsanity.com/paper/2306.06601/) (2023)

## Built on by (most cited first, 11 of 11)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning](https://mlsanity.com/paper/2508.10433/) (2025)
- [Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey](https://mlsanity.com/paper/2509.24322/) (2025)
- [OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs](https://mlsanity.com/paper/2503.21480/) (2025)
- [Why We Feel: Breaking Boundaries in Emotional Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2504.07521/) (2025)
- [When LLMs Team Up: The Emergence of Collaborative Affective Computing](https://mlsanity.com/paper/2506.01698/) (2025)
- [SCoPE: Shift-Aware Speaker-Conditioned Priors for Emotion Recognition in Conversations](https://mlsanity.com/paper/2607.20445/) (2026)
- [Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models](https://mlsanity.com/paper/2412.12606/) (2024)
- [MMD-Thinker: Adaptive Multi-Dimensional Thinking for Multimodal Misinformation Detection](https://mlsanity.com/paper/2511.13242/) (2025)
- [Mechanistic Decoding of Cognitive Constructs in Large Language Models](https://mlsanity.com/paper/2604.14593/) (2026)
- [NTDH: Complex Reasoning for Comprehensive Affective Analysis](https://mlsanity.com/paper/2608.06425/) (2026)

## Abstract (arXiv)

The field of emotion recognition of conversation (ERC) has been focusing on separating sentence feature encoding and context modeling, lacking exploration in generative paradigms based on unified designs. In this study, we propose a novel approach, InstructERC, to reformulate the ERC task from a discriminative framework to a generative framework based on Large Language Models (LLMs). InstructERC makes three significant contributions: (1) it introduces a simple yet effective retrieval template module, which helps the model explicitly integrate multi-granularity dialogue supervision information. (2) We introduce two additional emotion alignment tasks, namely speaker identification and emotion prediction tasks, to implicitly model the dialogue role relationships and future emotional tendencies in conversations. (3) Pioneeringly, we unify emotion labels across benchmarks through the feeling wheel to fit real application scenarios. InstructERC still perform impressively on this unified dataset. Our LLM-based plugin framework significantly outperforms all previous models and achieves comprehensive SOTA on three commonly used ERC datasets. Extensive analysis of parameter-efficient and data-scaling experiments provides empirical guidance for applying it in practical scenarios.