# Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

arXiv: https://arxiv.org/abs/2609.39049 · HTML: https://mlsanity.com/paper/2609.39049/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [Data set creation and empirical analysis for detecting signs of depression from social media postings](https://mlsanity.com/paper/2202.03047/) (2022)
- [Identifying Depressive Symptoms from Tweets: Figurative Language Enabled Multitask Learning Framework](https://mlsanity.com/paper/2011.06149/) (2020)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)
- [Interpretable Depression Detection from Social Media Text Using LLM-Derived Embeddings](https://mlsanity.com/paper/2506.06616/) (2025)
- [Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text](https://mlsanity.com/paper/2503.10095/) (2025)
- [Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset](https://mlsanity.com/paper/2403.00438/) (2024)
- [DepressionX: Knowledge Infused Residual Attention for Explainable Depression Severity Assessment](https://mlsanity.com/paper/2501.14985/) (2025)
- [Learning Evidence of Depression Symptoms via Prompt Induction](https://mlsanity.com/paper/2604.24376/) (2026)

## Abstract (arXiv)

A large language model (LLM) can rate depression severity directly from a social media post, or mark which clinical criteria the post shows and let a simple rule turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts as severe, whether prompted directly or with chain-of-thought, while the a priori rule, which uses no labels, beats both. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On one corpus, a relabeled stress dataset, a model using that dataset's own features, some computed from the post text, is not significantly different from frontier criteria extraction under the a priori rule.