# LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?

arXiv: https://arxiv.org/abs/2501.03624 · HTML: https://mlsanity.com/paper/2501.03624/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 16 of 16)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [The Capability of Large Language Models to Measure Psychiatric Functioning](https://mlsanity.com/paper/2308.01834/) (2023)
- [Do Large Language Models Align with Core Mental Health Counseling Competencies?](https://mlsanity.com/paper/2410.22446/) (2024)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classification](https://mlsanity.com/paper/2403.14255/) (2024)
- [WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensions](https://mlsanity.com/paper/2406.12058/) (2024)
- [Explainable Depression Symptom Detection in Social Media](https://mlsanity.com/paper/2310.13664/) (2023)
- [Aligning Large Language Models for Enhancing Psychiatric Interviews Through Symptom Delineation and Summarization: Pilot Study](https://mlsanity.com/paper/2403.17428/) (2024)
- [An Assessment on Comprehending Mental Health through Large Language Models](https://mlsanity.com/paper/2401.04592/) (2024)
- [Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments](https://mlsanity.com/paper/2405.11178/) (2024)
- [The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI](https://mlsanity.com/paper/2408.05977/) (2024)
- [Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges](https://mlsanity.com/paper/2409.17397/) (2024)

## Built on by (most cited first, 3 of 3)

- [AgentMental: An Interactive Multi-Agent Framework for Explainable and Adaptive Mental Health Assessment](https://mlsanity.com/paper/2508.11567/) (2025)
- [Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization](https://mlsanity.com/paper/2501.16106/) (2025)
- [Diagnosing Psychiatric Patients: Can Large Language and Machine Learning Models Perform Effectively in Emergency Cases?](https://mlsanity.com/paper/2509.00026/) (2025)

## Abstract (arXiv)

Large language models (LLMs) excel on many NLP benchmarks, but their behavior on real-world, semi-structured prediction remains underexplored. We present LlaMADRS, a benchmark for structured clinical assessment from dialogue built on the CAMI corpus of psychiatric interviews, comprising 5,804 expert annotations across 541 sessions. We evaluate 25 open-source models (standard and reasoning-augmented; 0.6B--400B parameters) and generate over 400,000 predictions. Our results demonstrate that strong open-source LLMs achieve item-level accuracy with residual error below clinically substantial thresholds. Additionally, an Item-then-Sum (ItS) strategy, assessing symptoms individually through discrete LLM calls before synthesizing final scores, significantly reduces error relative to Direct Total Score (DTS) prediction across most model architectures and scales, despite reasoning models attempting similar decomposition in the reasoning traces of their DTS predictions. In fact, we find that performance gains attributed to "reasoning" depend fundamentally on prompt design: standard models equipped with structured task definitions and examples match reasoning-augmented counterparts. Among the latter, longer reasoning traces correlate with reduced error; while higher model scale does across both architectures. Our results clarify when and why reasoning helps and offer actionable guidance for deploying LLMs in semi-structured clinical assessment.