mlsanityData to 14 Oct 2026

Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench

arXiv 2308.03656 ↗First posted 7 Aug 2023cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 47 of its 127 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

47 of 127 references.

References · 47

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  6. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  7. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  8. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  9. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  10. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  11. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  12. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
35 more
  1. 2023Personality Traits in Large Language Modelscited by 169
  2. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 147
  3. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  4. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  5. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  6. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 111
  7. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  8. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
  9. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  10. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  11. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 75
  12. 2022Automated Repair of Programs from Large Language Modelscited by 71
  13. 2023Can ChatGPT Assess Human Personalities? A General Evaluation Frameworkcited by 70
  14. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 67
  15. 2023Turning large language models into cognitive modelscited by 67
  16. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  17. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 65
  18. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  19. 2018On Testing Machine Learning Programscited by 48
  20. 2018Adversarial Sample Detection for Deep Neural Network through Model Mutation Testingcited by 48
  21. 2022Estimating the Personality of White-Box Language Modelscited by 48
  22. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  23. 2022Evaluating Psychological Safety of Large Language Modelscited by 26
  24. 2023CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 23
  25. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 22
  26. 2020Testing Machine Translation via Referential Transparencycited by 20
  27. 2022Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systemscited by 19
  28. 2019Testing DNN Image Classifiers for Confusion & Bias Errorscited by 18
  29. 2023Revisiting the Reliability of Psychological Scales on Large Language Modelscited by 16
  30. 2022AEON: A Method for Automatic Evaluation of NLP Test Casescited by 15
  31. 2023Towards Autonomous Testing Agents via Conversational Large Language Modelscited by 13
  32. 2020A Declarative Metamorphic Testing Framework for Autonomous Drivingcited by 10
  33. 2023Is GPT a Computational Model of Emotion? Detailed Analysiscited by 6
  34. 2022White-box Testing of NLP models with Mask Neuron Coveragecited by 2
  35. 2023Systematic Testing of the Data-Poisoning Robustness of KNNcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 14; 2026: 10.

54 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  4. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  5. 2024Large Language Models Meet NLP: A Surveycited by 103
  6. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  7. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
  8. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 53
  9. 2024HealMe: Harnessing Cognitive Reframing in Large Language Models for Psychotherapycited by 47
  10. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
  11. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  12. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
38 more
  1. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 25
  2. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  3. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  4. 2024Can LLMs make trade-offs involving stipulated pain and pleasure states?cited by 16
  5. 2024From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMscited by 13
  6. 2025Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflectioncited by 13
  7. 2025Decoding Emotion in the Deep: A Systematic Study of How LLMs Represent, Retain, and Express Emotioncited by 12
  8. 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
  9. 2025Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Designcited by 11
  10. 2023The Good, The Bad, and Why: Unveiling Emotions in Generative AIcited by 10
  11. 2023Affect Recognition in Conversations Using Large Language Modelscited by 9
  12. 2023The Confidence-Competence Gap in Large Language Models: A Cognitive Studycited by 9
  13. 2024Do Large Language Models Mirror Cognitive Language Processing?cited by 9
  14. 2023KLoB: a Benchmark for Assessing Knowledge Locating Methods in Language Modelscited by 8
  15. 2024Rethinking Emotion Annotations in the Era of Large Language Modelscited by 8
  16. 2025Modeling Earth-Scale Human-Like Societies with One Billion Agentscited by 8
  17. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  18. 2024A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Healthcited by 7
  19. 2024The Good, the Bad, and the Hulk-like GPT: Analyzing Emotional Decisions of Large Language Models in Cooperation and Bargaining Gamescited by 7
  20. 2025Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generationcited by 7
  21. 2026Gemma Needs Help: Investigating and Mitigating Emotional Instability in LLMscited by 7
  22. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Modelscited by 6
  23. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  24. 2026Latent Structure of Affective Representations in Large Language Modelscited by 4
  25. 2024GIEBench: Towards Holistic Evaluation of Group Identity-based Empathy for Large Language Modelscited by 3
  26. 2025Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMscited by 3
  27. 2024Can Generative Agents Predict Emotion?cited by 1
  28. 2024Red-Teaming for Inducing Societal Bias in Large Language Modelscited by 1
  29. 2024The Elephant in the Room -- Why AI Safety Demands Diverse Teamscited by 1
  30. 2025Incongruent Positivity: When Miscalibrated Positivity Undermines Online Supportive Conversationscited by 1
  31. 2025From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agentscited by 1
  32. 2025neuralFOMO: Can LLMs Handle Being Second Best? Measuring Envy-Like Preferences in Multi-Agent Settingscited by 1
  33. 2026MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisalcited by 1
  34. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  35. 2026EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbotscited by 1
  36. 2026ESC: Emotional Self-Correction for Reliable Vision-Language Modelscited by 1
  37. 2024Improving Emotional Support Delivery in Text-Based Community Safety Reporting Using Large Language Modelscited by 0
  38. 2024Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contextscited by 0

50 most cited of 54 All 54 (JSON)