Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Suicide prevention remains a critical public health challenge. While online platforms such as Reddit's r/SuicideWatch have historically provided spaces for individuals to express suicidal thoughts and seek community support, the advent of large language models (LLMs) introduces a new paradigm-where individuals may begin disclosing ideation to AI systems instead of humans. This study evaluates the capability of LLMs to perform automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS). We assess the zero-shot performance of six models-including Claude, GPT, Mistral, and LLaMA-in classifying posts across a 7-point severity scale (Levels 0-6). Results indicate that Claude and GPT closely align with human annotations, while Mistral achieves the lowest ordinal prediction error. Most models exhibit ordinal sensitivity, with misclassifications typically occurring between adjacent severity levels. We further analyze confusion patterns, misclassification sources, and ethical considerations, underscoring the importance of human oversight, transparency, and cautious deployment. Full code and supplementary materials are available at https://github.com/av9ash/llm_cssrs_code.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 11 of its 27 references that are in the map. First posted on arXiv 11 May 2025.
Builds on
11 of 27 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
- 2023Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuringcited by 53
- 2021Deep Learning for Suicide and Depression Identification with Unsupervised Label Correctioncited by 45
- 2019Suicidal Ideation Detection: A Review of Machine Learning Methods and Applicationscited by 31
- 2024Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Modelscited by 10
- 2017Neural Machine Translationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk
- 2025Evidence-Driven Marker Extraction for Social Media Suicide Risk Detection
- 2025Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation
- 2024A Dual-Prompting for Interpretable Mental Health Language Models
- 2026DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models
- 2025Systematic Evaluation of Machine-Generated Reasoning and PHQ-9 Labeling for Depression Detection Using Large Language Models
- 2026A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies
- 2025Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge