# Do Large Language Models Align with Core Mental Health Counseling Competencies?

arXiv: https://arxiv.org/abs/2410.22446 · HTML: https://mlsanity.com/paper/2410.22446/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 63 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding](https://mlsanity.com/paper/2305.12031/) (2023)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [Evaluation of ChatGPT for NLP-based Mental Health Applications](https://mlsanity.com/paper/2303.15727/) (2023)
- [Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches](https://mlsanity.com/paper/2404.14779/) (2024)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts](https://mlsanity.com/paper/2403.13786/) (2024)

## Built on by (most cited first, 19 of 19)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions](https://mlsanity.com/paper/2509.04183/) (2025)
- [FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health](https://mlsanity.com/paper/2509.14275/) (2025)
- [Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires](https://mlsanity.com/paper/2510.25384/) (2025)
- [LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?](https://mlsanity.com/paper/2501.03624/) (2025)
- [Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMs](https://mlsanity.com/paper/2508.16921/) (2025)
- [PsychAgent: An Experience-Driven Lifelong Learning Agent for Self-Evolving Psychological Counselor](https://mlsanity.com/paper/2604.00931/) (2026)
- [Responsible Evaluation of AI for Mental Health](https://mlsanity.com/paper/2602.00065/) (2026)
- [Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs](https://mlsanity.com/paper/2604.20382/) (2026)
- [From Reddit to Generative AI: Evaluating Large Language Models for Anxiety Support Fine-tuned on Social Media Data](https://mlsanity.com/paper/2505.18464/) (2025)
- [Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks](https://mlsanity.com/paper/2605.03472/) (2026)
- [Navigating the Rabbit Hole: Emergent Biases in LLM-Generated Attack Narratives Targeting Mental Health Groups](https://mlsanity.com/paper/2504.06160/) (2025)
- [MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory](https://mlsanity.com/paper/2508.07279/) (2025)
- [MotionTeller: Multi-modal Integration of Wearable Time-Series with LLMs for Health and Behavioral Understanding](https://mlsanity.com/paper/2512.21506/) (2025)
- [Initial Risk Probing and Feasibility Testing of Glow: a Generative AI-Powered Dialectical Behavior Therapy Skills Coach for Substance Use Recovery and HIV Prevention](https://mlsanity.com/paper/2602.08121/) (2026)
- [Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour](https://mlsanity.com/paper/2606.18129/) (2026)
- [Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation](https://mlsanity.com/paper/2609.21075/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)
- [BiGraph-Diffuse: A Bidirectional Diffusion Language Model with Graph-Structured Retrieval For Mental Health Counseling](https://mlsanity.com/paper/2609.29519/) (2026)

## Abstract (arXiv)

The rapid evolution of Large Language Models (LLMs) presents a promising solution to the global shortage of mental health professionals. However, their alignment with essential counseling competencies remains underexplored. We introduce CounselingBench, a novel NCMHCE-based benchmark evaluating 22 general-purpose and medical-finetuned LLMs across five key competencies. While frontier models surpass minimum aptitude thresholds, they fall short of expert-level performance, excelling in Intake, Assessment & Diagnosis but struggling with Core Counseling Attributes and Professional Practice & Ethics. Surprisingly, medical LLMs do not outperform generalist models in accuracy, though they provide slightly better justifications while making more context-related errors. These findings highlight the challenges of developing AI for mental health counseling, particularly in competencies requiring empathy and nuanced reasoning. Our results underscore the need for specialized, fine-tuned models aligned with core mental health counseling competencies and supported by human oversight before real-world deployment. Code and data associated with this manuscript can be found at: https://github.com/cuongnguyenx/CounselingBench