mlsanityData to 14 Oct 2026

Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts

arXiv 2604.02713 ↗First posted 3 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Conversational AI is increasingly deployed in emotionally charged and ethically sensitive interactions. Previous research has primarily concentrated on emotional benchmarks or static safety checks, overlooking how alignment unfolds in evolving conversation. We explore the research question: what breakdowns arise when conversational agents confront emotionally and ethically sensitive behaviors, and how do these affect dialogue quality? To stress-test chatbot performance, we develop a persona-conditioned user simulator capable of engaging in multi-turn dialogue with psychological personas and staged emotional pacing. Our analysis reveals that mainstream models exhibit recurrent breakdowns that intensify as emotional trajectories escalate. We identify several common failure patterns, including affective misalignments, ethical guidance failures, and cross-dimensional trade-offs where empathy supersedes or undermines responsibility. We organize these patterns into a taxonomy and discuss the design implications, highlighting the necessity to maintain ethical coherence and affective sensitivity throughout dynamic interactions. The study offers the HCI community a new perspective on the diagnosis and improvement of conversational AI in value-sensitive and emotionally charged contexts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

32 of 52 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2020Recipes for building an open-domain chatbotcited by 593
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
20 more
  1. 2021Towards Emotional Support Dialog Systemscited by 299
  2. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  3. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 116
  4. 2023SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversationscited by 110
  5. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  6. 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
  7. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
  8. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 69
  9. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  10. 2024A Computational Framework for Behavioral Assessment of LLM Therapistscited by 62
  11. 2016A User Simulator for Task-Completion Dialoguescited by 57
  12. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  13. 2024ESC-Eval: Evaluating Emotion Support Conversations in Large Language Modelscited by 15
  14. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  15. 2025ESC-Judge: A Framework for Comparing Emotional Support Conversational Agentscited by 8
  16. 2025Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systemscited by 7
  17. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arenacited by 6
  18. 2025Tailored Emotional LLM-Supporter: Enhancing Cultural Sensitivitycited by 5
  19. 2025Does Your AI Agent Get You? A Personalizable Framework for Approximating Human Models from Argumentation-based Dialogue Tracescited by 3
  20. 2024VCounselor: A Psychological Intervention Chat Agent Based on a Knowledge-Enhanced Large Language Modelcited by 1

Built on by

Not cited yet.