# MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models

arXiv: https://arxiv.org/abs/2502.15418 · HTML: https://mlsanity.com/paper/2502.15418/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts](https://mlsanity.com/paper/2207.04674/) (2022)
- [Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health](https://mlsanity.com/paper/2304.10447/) (2023)
- [PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models](https://mlsanity.com/paper/2311.09189/) (2023)
- [Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis](https://mlsanity.com/paper/2410.03908/) (2024)
- [MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcare](https://mlsanity.com/paper/2405.12619/) (2024)
- [ConvCounsel: A Conversational Dataset for Student Counseling](https://mlsanity.com/paper/2411.00604/) (2024)

## Built on by (most cited first, 7 of 7)

- [CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering](https://mlsanity.com/paper/2506.08584/) (2025)
- [MiraMind: Benchmarking Reliable Mental Health Reasoning beyond Answer Accuracy](https://mlsanity.com/paper/2512.09636/) (2025)
- [OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs](https://mlsanity.com/paper/2603.25105/) (2026)
- [An Offline Mobile Conversational Agent for Mental Health Support: Learning from Emotional Dialogues and Psychological Texts with Student-Centered Evaluation](https://mlsanity.com/paper/2507.10580/) (2025)
- [Hierarchical Dual-Strategy Unlearning for Biomedical and Healthcare Intelligence Using Imperfect and Privacy-Sensitive Medical Data](https://mlsanity.com/paper/2511.19498/) (2025)
- [Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour](https://mlsanity.com/paper/2606.18129/) (2026)
- [CANDI: Contextual Alignment for Niche Domains Question Answering](https://mlsanity.com/paper/2607.11891/) (2026)

## Abstract (arXiv)

Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common. Large Language Models (LLMs) have seen a vast application in healthcare, specifically in answering medical questions. However, there is a lack of standard benchmarking datasets for question answering (QA) in mental health. Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs). Previous mental health datasets have focused primarily on text classification into specific labels or disorders. MHQA, on the other hand, presents question-answering for mental health focused on four key domains: anxiety, depression, trauma, and obsessive/compulsive issues, with diverse question types, namely, factoid, diagnostic, prognostic, and preventive. We use PubMed abstracts as the primary source for QA. We develop a rigorous pipeline for LLM-based identification of information from abstracts based on various selection criteria and converting it into QA pairs. Further, valid QA pairs are extracted based on post-hoc validation criteria. Overall, our MHQA dataset consists of 2,475 expert-verified gold standard instances called MHQA-gold and ~56.1k pairs pseudo labeled using external medical references. We report F1 scores on different LLMs along with few-shot and supervised fine-tuning experiments, further discussing the insights for the scores.