# Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation

arXiv: https://arxiv.org/abs/2507.14693 · HTML: https://mlsanity.com/paper/2507.14693/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 28 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 3 of 3)

- [Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networks](https://mlsanity.com/paper/2004.07601/) (2020)
- [Pearson-Matthews correlation coefficients for binary and multinary classification and hypothesis testing](https://mlsanity.com/paper/2305.05974/) (2023)
- [Data Quality Matters: Suicide Intention Detection on Social Media Posts Using RoBERTa-CNN](https://mlsanity.com/paper/2402.02262/) (2024)

## Abstract (arXiv)

Suicidal ideation detection is critical for real-time suicide prevention, yet its progress faces two under-explored challenges: limited language coverage and unreliable annotation practices. Most available datasets are in English, but even among these, high-quality, human-annotated data remains scarce. As a result, many studies rely on available pre-labeled datasets without examining their annotation process or label reliability. The lack of datasets in other languages further limits the global realization of suicide prevention via artificial intelligence (AI). In this study, we address one of these gaps by constructing a novel Turkish suicidal ideation corpus derived from social media posts and introducing a resource-efficient annotation framework involving three human annotators and two large language models (LLMs). We then address the remaining gaps by performing a bidirectional evaluation of label reliability and model consistency across this dataset and three popular English suicidal ideation detection datasets, using transfer learning through eight pre-trained sentiment and emotion classifiers. These transformers help assess annotation consistency and benchmark model performance against manually labeled data. Our findings underscore the need for more rigorous, language-inclusive approaches to annotation and evaluation in mental health natural language processing (NLP) while demonstrating the questionable performance of popular models with zero-shot transfer learning. We advocate for transparency in model training and dataset construction in mental health NLP, prioritizing data and model reliability.