mlsanityData to 7 Oct 2026

EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

arXiv 2504.09689 ↗First posted 13 Apr 2025cs.AI cs.CL cs.CYcited by 21

Not on the map. Why?

Abstract (arXiv)

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: EmoEval simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions. Our code is available at: https://github.com/1akaman/EmoAgent

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 37 of its 69 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

37 of 69 references.

References · 37

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  5. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  6. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 208
  7. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  8. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  9. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  10. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  11. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  12. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Modelscited by 61
25 more
  1. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  2. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  3. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  4. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 42
  5. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  6. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 35
  7. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  8. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  9. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  10. 2024LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunitiescited by 27
  11. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  12. 2024Can AI Relate: Testing Large Language Model Response for Mental Health Supportcited by 21
  13. 2024The Better Angels of Machine Personality: How Personality Relates to LLM Safetycited by 21
  14. 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agentscited by 21
  15. 2024PATIENT-Psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionalscited by 19
  16. 2023An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectivescited by 16
  17. 2024Rapid Response: Mitigating LLM Jailbreaks with a Few Examplescited by 15
  18. 2024PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Gamescited by 10
  19. 2024RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Processcited by 10
  20. 2024Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Toolscited by 9
  21. 2024Characteristic AI Agents via Large Language Modelscited by 5
  22. 2024A Framework for Evaluating Appropriateness, Trustworthiness, and Safety in Mental Wellness AI Chatbotscited by 5
  23. 2024PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurementscited by 5
  24. 2025A Layered Multi-Expert Framework for Long-Context Mental Health Assessmentscited by 4
  25. 2024Dr. GPT in Campus Counseling: Understanding Higher Education Students' Opinions on LLM-assisted Mental Health Servicescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  2. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  3. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 14
  4. 2025Trustless Autonomy: Understanding Motivations, Benefits, and Governance Dilemmas in Self-Sovereign Decentralized AI Agentscited by 10
  5. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  6. 2025Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judgecited by 4
  7. 2025Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMscited by 3
  8. 2026PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethicscited by 3
  9. 2026Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teamingcited by 3
  10. 2026Responsible Evaluation of AI for Mental Healthcited by 2
  11. 2026Understanding Risk and Dependency in AI Chatbot Use from User Discoursecited by 2
  12. 2026A Safety-Aware Role-Orchestrated Multi-Agent LLM Framework for Behavioral Health Communication Simulationcited by 1
9 more
  1. 2026AI Safety Training Can be Clinically Harmfulcited by 1
  2. 2025Systematizing LLM Persona Design: A Four-Quadrant Technical Taxonomy for AI Companion Applicationscited by 0
  3. 2025When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Modelscited by 0
  4. 2026TherapyProbe: Generating Design Knowledge for Relational Safety in Mental Health Chatbots Through Adversarial Simulationcited by 0
  5. 2026Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbotscited by 0
  6. 2026Trust Stack for Mental Health AI: A Survey of Calibration across Human, Interaction, and AI Layerscited by 0
  7. 2026DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbotscited by 0
  8. 2026How Should I Pick a Foundation Model for My Robot? In Favor of a Community Evaluation Framework for Social Robotscited by 0
  9. 2026From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Healthcited by 0