# MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance

arXiv: https://arxiv.org/abs/2503.13509 · HTML: https://mlsanity.com/paper/2503.13509/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 76 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Semantic Answer Similarity for Evaluating Question Answering Models](https://mlsanity.com/paper/2108.06130/) (2021)
- [CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering](https://mlsanity.com/paper/2403.16008/) (2024)
- [Large Language Models Produce Responses Perceived to be Empathic](https://mlsanity.com/paper/2403.18148/) (2024)
- [Speaker and Time-aware Joint Contextual Learning for Dialogue-act Classification in Counselling Conversations](https://mlsanity.com/paper/2111.06647/) (2021)
- [Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals](https://mlsanity.com/paper/2308.15192/) (2023)
- [Counseling Summarization using Mental Health Knowledge Guided Utterance Filtering](https://mlsanity.com/paper/2206.03886/) (2022)
- [Harnessing Large Language Models' Empathetic Response Generation Capabilities for Online Mental Health Counselling Support](https://mlsanity.com/paper/2310.08017/) (2023)
- [Clinical Evidence Engine: Proof-of-Concept For A Clinical-Domain-Agnostic Decision Support Infrastructure](https://mlsanity.com/paper/2111.00621/) (2021)

## Built on by (most cited first, 30 of 39)

- [When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation](https://mlsanity.com/paper/2510.19032/) (2025)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions](https://mlsanity.com/paper/2509.04183/) (2025)
- [PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry](https://mlsanity.com/paper/2509.09711/) (2025)
- [Assessing the Quality of Mental Health Support in LLM Responses through Multi-Attribute Human Evaluation](https://mlsanity.com/paper/2601.18630/) (2026)
- [PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions](https://mlsanity.com/paper/2509.07370/) (2025)
- [Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbots](https://mlsanity.com/paper/2509.16444/) (2025)
- [LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis](https://mlsanity.com/paper/2602.09379/) (2026)
- [A Comprehensive Review of Datasets for Clinical Mental Health AI Systems](https://mlsanity.com/paper/2508.09809/) (2025)
- [CALM-IT: Enhancing Synthetic Motivational Interviewing Dialogue Quality with Conversational Dynamics](https://mlsanity.com/paper/2601.10085/) (2026)
- [MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs](https://mlsanity.com/paper/2602.06268/) (2026)
- [Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AI](https://mlsanity.com/paper/2603.07466/) (2026)
- [MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models](https://mlsanity.com/paper/2604.17730/) (2026)
- [T-FIX: Text-Based Explanations with Features Interpretable to eXperts](https://mlsanity.com/paper/2511.04070/) (2025)
- [Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs](https://mlsanity.com/paper/2604.20382/) (2026)
- [RHealthTwin: Towards Responsible and Multimodal Digital Twins for Personalized Well-being](https://mlsanity.com/paper/2506.08486/) (2025)
- [EthicsMH: A Pilot Benchmark for Ethical Reasoning in Mental Health AI](https://mlsanity.com/paper/2509.11648/) (2025)
- [EFT-CoT: A Multi-Agent Chain-of-Thought Framework for Emotion-Focused Therapy](https://mlsanity.com/paper/2601.17842/) (2026)
- [OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs](https://mlsanity.com/paper/2603.25105/) (2026)
- [Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs](https://mlsanity.com/paper/2604.07655/) (2026)
- [Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations](https://mlsanity.com/paper/2605.15205/) (2026)
- [MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters](https://mlsanity.com/paper/2607.08257/) (2026)
- [Privacy-Preserving Split Learning for Federated LLM Fine-Tuning](https://mlsanity.com/paper/2609.09794/) (2026)
- [Mentalic Net: Development of RAG-based Conversational AI and Evaluation Framework for Mental Health Support](https://mlsanity.com/paper/2509.04456/) (2025)
- [Plural Voices, Single Agent: Towards Inclusive AI in Multi-User Domestic Spaces](https://mlsanity.com/paper/2510.19008/) (2025)
- [InvisibleBench: A Deployment Gate for Caregiving Relationship AI](https://mlsanity.com/paper/2511.20733/) (2025)
- [VERA-MH: Validation of Ethical and Responsible AI in Mental Health](https://mlsanity.com/paper/2605.13318/) (2026)
- [EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation](https://mlsanity.com/paper/2605.22286/) (2026)
- [Supervised Learning Objectives for High-Dimensional Treatment Effects](https://mlsanity.com/paper/2605.27281/) (2026)
- [Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm](https://mlsanity.com/paper/2605.27914/) (2026)

## Abstract (arXiv)

We introduce MentalChat16K, an English benchmark dataset combining a synthetic mental health counseling dataset and a dataset of anonymized transcripts from interventions between Behavioral Health Coaches and Caregivers of patients in palliative or hospice care. Covering a diverse range of conditions like depression, anxiety, and grief, this curated dataset is designed to facilitate the development and evaluation of large language models for conversational mental health assistance. By providing a high-quality resource tailored to this critical domain, MentalChat16K aims to advance research on empathetic, personalized AI solutions to improve access to mental health support services. The dataset prioritizes patient privacy, ethical considerations, and responsible data usage. MentalChat16K presents a valuable opportunity for the research community to innovate AI technologies that can positively impact mental well-being. The dataset is available at https://huggingface.co/datasets/ShenLab/MentalChat16K and the code and documentation are hosted on GitHub at https://github.com/ChiaPatricia/MentalChat16K.