# Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal Risks in Chinese Social Media

arXiv: https://arxiv.org/abs/2309.03564 · HTML: https://mlsanity.com/paper/2309.03564/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 78 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction](https://mlsanity.com/paper/2305.02466/) (2023)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2112.12731/) (2021)
- [LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation](https://mlsanity.com/paper/2305.13614/) (2023)
- [Latent Suicide Risk Detection on Microblog via Suicide-Oriented Word Embeddings and Layered Attention](https://mlsanity.com/paper/1910.12038/) (2019)
- [Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals](https://mlsanity.com/paper/2308.15192/) (2023)
- [Read, Diagnose and Chat: Towards Explainable and Interactive LLMs-Augmented Depression Detection in Social Media](https://mlsanity.com/paper/2305.05138/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Large Language Models in Mental Health Care: a Scoping Review](https://mlsanity.com/paper/2401.02984/) (2024)
- [Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts](https://mlsanity.com/paper/2403.13786/) (2024)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents](https://mlsanity.com/paper/2504.02800/) (2025)
- [KoACD: The First Korean Adolescent Dataset for Cognitive Distortion Analysis via Role-Switching Multi-LLM Negotiation](https://mlsanity.com/paper/2505.00367/) (2025)
- [Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge](https://mlsanity.com/paper/2508.08236/) (2025)
- [SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis](https://mlsanity.com/paper/2404.12659/) (2024)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

On social media, users often express their personal feelings, which may exhibit cognitive distortions or even suicidal tendencies on certain specific topics. Early recognition of these signs is critical for effective psychological intervention. In this paper, we introduce two novel datasets from Chinese social media: SOS-HL-1K for suicidal risk classification and SocialCD-3K for cognitive distortions detection. The SOS-HL-1K dataset contained 1,249 posts and SocialCD-3K dataset was a multi-label classification dataset that containing 3,407 posts. We propose a comprehensive evaluation using two supervised learning methods and eight large language models (LLMs) on the proposed datasets. From the prompt engineering perspective, we experimented with two types of prompt strategies, including four zero-shot and five few-shot strategies. We also evaluated the performance of the LLMs after fine-tuning on the proposed tasks. The experimental results show that there is still a huge gap between LLMs relying only on prompt engineering and supervised learning. In the suicide classification task, this gap is 6.95% points in F1-score, while in the cognitive distortion task, the gap is even more pronounced, reaching 31.53% points in F1-score. However, after fine-tuning, this difference is significantly reduced. In the suicide and cognitive distortion classification tasks, the gap decreases to 4.31% and 3.14%, respectively. This research highlights the potential of LLMs in psychological contexts, but supervised learning remains necessary for more challenging tasks. All datasets and code are made available.