mlsanityData to 7 Oct 2026

PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models

arXiv 2311.09189 ↗First posted 15 Nov 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Evaluating Large Language Models (LLMs) in the mental health domain poses distinct challenged from other domains, given the subtle and highly subjective nature of symptoms that exhibit significant variability among individuals. This paper presents PsyEval, the first comprehensive suite of mental health-related tasks for evaluating LLMs. PsyEval encompasses five sub-tasks that evaluate three critical dimensions of mental health. This comprehensive framework is designed to thoroughly assess the unique challenges and intricacies of mental health-related tasks, making PsyEval a highly specialized and valuable tool for evaluating LLM performance in this domain. We evaluate twelve advanced LLMs using PsyEval. Experiment results not only demonstrate significant room for improvement in current LLMs concerning mental health but also unveil potential directions for future model optimization.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 25 of its 56 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

25 of 56 references.

References · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
13 more
  1. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  2. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  3. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  4. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  5. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  6. 2021PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Supportcited by 92
  7. 2018SMHD: A Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditionscited by 73
  8. 2023Evaluation of ChatGPT for NLP-based Mental Health Applicationscited by 53
  9. 2021Deep Learning for Suicide and Depression Identification with Unsupervised Label Correctioncited by 45
  10. 2023ChatCounselor: A Large Language Models for Mental Health Supportcited by 44
  11. 2023Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPTcited by 41
  12. 2022D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chatcited by 37
  13. 2023A Benchmark for Understanding Dialogue Safety in Mental Health Supportcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applicationscited by 7
  2. 2025MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Modelscited by 7
  3. 2025Reasoning Is Not All You Need: Examining LLMs for Multi-Turn Mental Health Conversationscited by 6
  4. 2024An Assessment on Comprehending Mental Health through Large Language Modelscited by 5
  5. 2025Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judgecited by 4
  6. 2024A Multi-Agent Dual Dialogue System to Support Mental Health Care Providerscited by 1
  7. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  8. 2026TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Healthcited by 1
  9. 2026TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbotscited by 1
  10. 2026EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbotscited by 1
  11. 2025Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Supportcited by 0
  12. 2026Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviourcited by 0
1 more
  1. 2026The methodology of Constructing the Large-Scale Dataset for Detecting Presuicidal and Anti-Suicidal Signals in Social Media Texts in Russiancited by 0