Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor
On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment
Abstract (arXiv)
Chatbot research is advancing with the growing importance of chatbots in fields that require human interactions, such as customer support and mental health care. Despite these advancements, chatbots still face significant challenges in understanding subtle nuances and managing long conversation histories. To address these issues, our study introduces a dual approach: firstly, we employ Emotional Preference Optimization (EPO) to train chatbots not only with correct responses but also with counter-emotional responses-those that are contextually similar but emotionally divergent. This training enables the model to discern fine nuance distinctions between correct and counter-emotional responses, thereby enhancing the quality of its responses. Secondly, we introduce MambaCompressor to effectively compress and manage extensive conversation histories, significantly reducing time and memory complexities while improving the chatbot's contextual understanding. Our comprehensive experiments across multiple datasets demonstrate that our model significantly outperforms existing models in generating empathetic responses and efficiently managing lengthy dialogues.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 60 references that are in the map. First posted on arXiv 23 Dec 2024.
Builds on
34 of 60 references.
References · 34
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
22 more
- 2024Qwen2 Technical Reportcited by 2,120
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
- 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
- 2023VideoChat: Chat-Centric Video Understandingcited by 857
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
- 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
- 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
- 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
- 2024Token-level Direct Preference Optimizationcited by 135
- 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 89
- 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
- 2024VideoMamba: Spatio-Temporal Selective State Space Modelcited by 25
- 2024Towards Multimodal Emotional Support Conversation Systemscited by 23
- 2024EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbotcited by 17
- 2021ERICA: An Empathetic Android Companion for Covid-19 Quarantinecited by 8
- 2024ReflectDiffu:Reflect between Emotion-intent Contagion and Mimicry for Empathetic Response Generation via a RL-Diffusion Frameworkcited by 6
- 2021Nora: The Well-Being Coachcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization
- 2024ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modeling
- 2026SELF-EMO: Emotional Self-Evolution from Recognition to Consistent Expression
- 2025From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agents
- 2024AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues
- 2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Models
- 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
- 2025PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning