# Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media

arXiv: https://arxiv.org/abs/2305.16797 · HTML: https://mlsanity.com/paper/2305.16797/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 91 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences](https://mlsanity.com/paper/1804.07000/) (2018)
- [DeepSleepNet-Lite: A Simplified Automatic Sleep Stage Scoring Model with Uncertainty Estimates](https://mlsanity.com/paper/2108.10600/) (2021)
- [Explainable Identification of Dementia from Transcripts using Transformer Networks](https://mlsanity.com/paper/2109.06980/) (2021)
- [Understanding and Measuring Psychological Stress using Social Media](https://mlsanity.com/paper/1811.07430/) (2018)
- [DepressionNet: A Novel Summarization Boosted Deep Framework for Depression Detection on Social Media](https://mlsanity.com/paper/2105.10878/) (2021)
- [Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision](https://mlsanity.com/paper/1805.12307/) (2018)
- [Complaint Identification in Social Media with Transformer Networks](https://mlsanity.com/paper/2010.10910/) (2020)
- [Detecting "Smart" Spammers On Social Network: A Topic Model Approach](https://mlsanity.com/paper/1604.08504/) (2016)

## Built on by (most cited first, 12 of 12)

- [PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation](https://mlsanity.com/paper/2407.05721/) (2024)
- [On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlook](https://mlsanity.com/paper/2410.08793/) (2024)
- [Multitask learning for recognizing stress and depression in social media](https://mlsanity.com/paper/2305.18907/) (2023)
- [Harnessing the Power of Hugging Face Transformers for Predicting Mental Health Disorders in Social Networks](https://mlsanity.com/paper/2306.16891/) (2023)
- [Multimodal Detection of Bots on X (Twitter) using Transformers](https://mlsanity.com/paper/2308.14484/) (2023)
- [Sentiment Informed Sentence BERT-Ensemble Algorithm for Depression Detection](https://mlsanity.com/paper/2409.13713/) (2024)
- [Depression detection in social media posts using transformer-based models and auxiliary features](https://mlsanity.com/paper/2409.20048/) (2024)
- [On-device Federated Learning in Smartphones for Detecting Depression from Reddit Posts](https://mlsanity.com/paper/2410.13709/) (2024)
- [Detecting anxiety and depression in dialogues: a multi-label and explainable approach](https://mlsanity.com/paper/2412.17651/) (2024)
- [A Graph Neural Architecture Search Approach for Identifying Bots in Social Media](https://mlsanity.com/paper/2411.16285/) (2024)
- [MoPHES:Leveraging on-device LLMs as Agent for Mobile Psychological Health Evaluation and Support](https://mlsanity.com/paper/2510.16085/) (2025)
- [MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection](https://mlsanity.com/paper/2606.12649/) (2026)

## Abstract (arXiv)

In today's fast-paced world, the rates of stress and depression present a surge. Social media provide assistance for the early detection of mental health conditions. Existing methods mainly introduce feature extraction approaches and train shallow machine learning classifiers. Other researches use deep neural networks or transformers. Despite the fact that transformer-based models achieve noticeable improvements, they cannot often capture rich factual knowledge. Although there have been proposed a number of studies aiming to enhance the pretrained transformer-based models with extra information or additional modalities, no prior work has exploited these modifications for detecting stress and depression through social media. In addition, although the reliability of a machine learning model's confidence in its predictions is critical for high-risk applications, there is no prior work taken into consideration the model calibration. To resolve the above issues, we present the first study in the task of depression and stress detection in social media, which injects extra linguistic information in transformer-based models, namely BERT and MentalBERT. Specifically, the proposed approach employs a Multimodal Adaptation Gate for creating the combined embeddings, which are given as input to a BERT (or MentalBERT) model. For taking into account the model calibration, we apply label smoothing. We test our proposed approaches in three publicly available datasets and demonstrate that the integration of linguistic features into transformer-based models presents a surge in the performance. Also, the usage of label smoothing contributes to both the improvement of the model's performance and the calibration of the model. We finally perform a linguistic analysis of the posts and show differences in language between stressful and non-stressful texts, as well as depressive and non-depressive posts.