mlsanityData to 14 Oct 2026

CARE-MH: Towards Unified, Reproducible, and Comparable Evaluation of Mental Health LLMs

arXiv 2607.24754 ↗First posted 26 May 2026cs.HC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used to provide mental health support, requiring reliable evaluation of safety, empathy, and therapeutic appropriateness. However, existing mental health benchmarks are difficult to reproduce and compare due to inconsistent evaluation designs and metric definitions. We present CARE-MH, a unified framework for comparable and reproducible evaluation of mental health LLMs. Using CARE-MH, we reproduce and analyze state-of-the-art benchmarks, revealing that reproducibility depends strongly on model stability and that cross-benchmark disagreement primarily arises from differences in metric definitions. Our findings highlight the need for standardized evaluation configurations and shared metric definitions for future mental health LLM benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 79 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

16 of 79 references.

References · 16

  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  4. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  5. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  6. 2024The opportunities and risks of large language models in mental healthcited by 54
  7. 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
  8. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 21
  9. 2025When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluationcited by 19
  10. 2023Which is better? Exploring Prompting Strategy For LLM-based Metricscited by 14
  11. 2024Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Stylecited by 11
  12. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 10
4 more
  1. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillationcited by 7
  2. 2024Script-Strategy Aligned Generation: Aligning LLMs with Expert-Crafted Dialogue Scripts and Therapeutic Strategies for Psychotherapycited by 7
  3. 2025PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatrycited by 6
  4. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5

Built on by

Not cited yet.