# Mitigating Unhelpfulness in Emotional Support Conversations with Multifaceted AI Feedback

arXiv: https://arxiv.org/abs/2401.05928 · HTML: https://mlsanity.com/paper/2401.05928/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Challenges in Building Intelligent Open-domain Dialog Systems](https://mlsanity.com/paper/1905.05709/) (2019)
- [MISC: A MIxed Strategy-Aware Model Integrating COMET for Emotional Support Conversation](https://mlsanity.com/paper/2203.13560/) (2022)
- [Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning](https://mlsanity.com/paper/2210.04242/) (2022)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations](https://mlsanity.com/paper/2305.10172/) (2023)
- [Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning](https://mlsanity.com/paper/2306.03350/) (2023)
- [Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability](https://mlsanity.com/paper/2106.07393/) (2021)
- [TransESC: Smoothing Emotional Support Conversation via Turn-Level State Transition](https://mlsanity.com/paper/2305.03296/) (2023)
- [Aligning Language Models with Human Preferences via a Bayesian Approach](https://mlsanity.com/paper/2310.05782/) (2023)
- [Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization](https://mlsanity.com/paper/2108.11846/) (2021)
- [Empathetic Response Generation through Graph-based Multi-hop Reasoning on Emotional Causality](https://mlsanity.com/paper/2110.04614/) (2021)

## Built on by (most cited first, 5 of 5)

- [SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent](https://mlsanity.com/paper/2412.08389/) (2024)
- [From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment](https://mlsanity.com/paper/2505.16610/) (2025)
- [Incongruent Positivity: When Miscalibrated Positivity Undermines Online Supportive Conversations](https://mlsanity.com/paper/2509.10184/) (2025)
- [LifeSide: Benchmarking Agents as Lifelong Digital Companions](https://mlsanity.com/paper/2606.04660/) (2026)
- [FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation](https://mlsanity.com/paper/2610.20051/) (2026)

## Abstract (arXiv)

An emotional support conversation system aims to alleviate users' emotional distress and assist them in addressing their challenges. To generate supportive responses, it is critical to consider multiple factors such as empathy, support strategies, and response coherence, as established in prior methods. Nonetheless, previous models occasionally generate unhelpful responses, which intend to provide support but display counterproductive effects. According to psychology and communication theories, poor performance in just one contributing factor might cause a response to be unhelpful. From the model training perspective, since these models have not been exposed to unhelpful responses during their training phase, they are unable to distinguish if the tokens they generate might result in unhelpful responses during inference. To address this issue, we introduce a novel model-agnostic framework named mitigating unhelpfulness with multifaceted AI feedback for emotional support (Muffin). Specifically, Muffin employs a multifaceted AI feedback module to assess the helpfulness of responses generated by a specific model with consideration of multiple factors. Using contrastive learning, it then reduces the likelihood of the model generating unhelpful responses compared to the helpful ones. Experimental results demonstrate that Muffin effectively mitigates the generation of unhelpful responses while slightly increasing response fluency and relevance.