# Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset

arXiv: https://arxiv.org/abs/2403.00438 · HTML: https://mlsanity.com/paper/2403.00438/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://mlsanity.com/paper/1709.01848/) (2017)
- [Identifying Depressive Symptoms from Tweets: Figurative Language Enabled Multitask Learning Framework](https://mlsanity.com/paper/2011.06149/) (2020)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)

## Built on by (most cited first, 3 of 3)

- [Datasets for Depression Modeling in Social Media: An Overview](https://mlsanity.com/paper/2503.21513/) (2025)
- [Evaluating Lexicon Incorporation for Depression Symptom Estimation](https://mlsanity.com/paper/2404.19359/) (2024)
- [Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity](https://mlsanity.com/paper/2609.39049/) (2026)

## Abstract (arXiv)

This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.