# The Capability of Large Language Models to Measure Psychiatric Functioning

arXiv: https://arxiv.org/abs/2308.01834 · HTML: https://mlsanity.com/paper/2308.01834/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 30 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)

## Built on by (most cited first, 26 of 26)

- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [Towards a Personal Health Large Language Model](https://mlsanity.com/paper/2406.06474/) (2024)
- [Can AI Relate: Testing Large Language Model Response for Mental Health Support](https://mlsanity.com/paper/2405.12021/) (2024)
- [Transforming Wearable Data into Personal Health Insights using Large Language Model Agents](https://mlsanity.com/paper/2406.06464/) (2024)
- [MAGI: Multi-Agent Guided Interview for Psychiatric Assessment](https://mlsanity.com/paper/2504.18260/) (2025)
- [LLM Questionnaire Completion for Automatic Psychiatric Assessment](https://mlsanity.com/paper/2406.06636/) (2024)
- [A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions](https://mlsanity.com/paper/2502.11095/) (2025)
- [LLMs-Healthcare : Current Applications and Challenges of Large Language Models in various Medical Specialties](https://mlsanity.com/paper/2311.12882/) (2023)
- [Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safety](https://mlsanity.com/paper/2506.12482/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Aligning Large Language Models for Enhancing Psychiatric Interviews Through Symptom Delineation and Summarization: Pilot Study](https://mlsanity.com/paper/2403.17428/) (2024)
- [An Assessment on Comprehending Mental Health through Large Language Models](https://mlsanity.com/paper/2401.04592/) (2024)
- [Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapy](https://mlsanity.com/paper/2410.05824/) (2024)
- [Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments](https://mlsanity.com/paper/2405.11178/) (2024)
- [Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ](https://mlsanity.com/paper/2407.19340/) (2024)
- [Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance](https://mlsanity.com/paper/2412.10417/) (2024)
- [LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?](https://mlsanity.com/paper/2501.03624/) (2025)
- [Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming](https://mlsanity.com/paper/2602.19948/) (2026)
- [Explaining GPTs' Schema of Depression: A Machine Behavior Analysis](https://mlsanity.com/paper/2411.13800/) (2024)
- [Evaluating Large Language Models for Anxiety, Depression, and Stress Detection: Insights into Prompting Strategies and Synthetic Data](https://mlsanity.com/paper/2511.07044/) (2025)
- [Navigating the Rabbit Hole: Emergent Biases in LLM-Generated Attack Narratives Targeting Mental Health Groups](https://mlsanity.com/paper/2504.06160/) (2025)
- [Using Large Language Models to Measure Symptom Severity in Patients At Risk for Schizophrenia](https://mlsanity.com/paper/2508.10226/) (2025)
- [Language-model ratings of depression reflect the rater more than the patient](https://mlsanity.com/paper/2610.08501/) (2026)

## Abstract (arXiv)

The current work investigates the capability of Large language models (LLMs) that are explicitly trained on large corpuses of medical knowledge (Med-PaLM 2) to predict psychiatric functioning from patient interviews and clinical descriptions without being trained to do so. To assess this, n = 145 depression and n =115 PTSD assessments and n = 46 clinical case studies across high prevalence/high comorbidity disorders (Depressive, Anxiety, Psychotic, trauma and stress, Addictive disorders) were analyzed using prompts to extract estimated clinical scores and diagnoses. Results demonstrate that Med-PaLM 2 is capable of assessing psychiatric functioning across a range of psychiatric conditions with the strongest performance being the prediction of depression scores based on standardized assessments (Accuracy range= 0.80 - 0.84) which were statistically indistinguishable from human clinical raters t(1,144) = 1.20; p = 0.23. Results show the potential for general clinical language models to flexibly predict psychiatric risk based on free descriptions of functioning from both patients and clinicians.