# Can We Assess Mental Health through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation

arXiv: https://arxiv.org/abs/1807.07351 · HTML: https://mlsanity.com/paper/1807.07351/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2018.

## Built on by (most cited first, 6 of 6)

- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [The Rise of Small Language Models in Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2504.17119/) (2025)
- [Identifying Moments of Change from Longitudinal User Text](https://mlsanity.com/paper/2205.05593/) (2022)
- [Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses](https://mlsanity.com/paper/2206.11155/) (2022)
- [Creation and evaluation of timelines for longitudinal user posts](https://mlsanity.com/paper/2303.05891/) (2023)
- [The Problem of Semantic Shift in Longitudinal Monitoring of Social Media: A Case Study on Mental Health During the COVID-19 Pandemic](https://mlsanity.com/paper/2206.11160/) (2022)

## Abstract (arXiv)

Predicting mental health from smartphone and social media data on a longitudinal basis has recently attracted great interest, with very promising results being reported across many studies. Such approaches have the potential to revolutionise mental health assessment, if their development and evaluation follows a real world deployment setting. In this work we take a closer look at state-of-the-art approaches, using different mental health datasets and indicators, different feature sources and multiple simulations, in order to assess their ability to generalise. We demonstrate that under a pragmatic evaluation framework, none of the approaches deliver or even approach the reported performances. In fact, we show that current state-of-the-art approaches can barely outperform the most na"ive baselines in the real-world setting, posing serious questions not only about their deployment ability, but also about the contribution of the derived features for the mental health assessment task and how to make better use of such data in the future.