# A Multimodal Framework for Depression Detection during Covid-19 via Harvesting Social Media: A Novel Dataset and Method

arXiv: https://arxiv.org/abs/2511.00424 · HTML: https://mlsanity.com/paper/2511.00424/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 1 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Empath: Understanding Topic Signals in Large-Scale Text](https://mlsanity.com/paper/1602.06979/) (2016)
- [Sentiment of Emojis](https://mlsanity.com/paper/1509.07761/) (2015)
- [It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers](https://mlsanity.com/paper/2301.05453/) (2023)
- [Understanding and Measuring Psychological Stress using Social Media](https://mlsanity.com/paper/1811.07430/) (2018)
- [Adapting Deep Learning Methods for Mental Health Prediction on Social Media](https://mlsanity.com/paper/2003.07634/) (2020)
- [Detecting Community Depression Dynamics Due to COVID-19 Pandemic in Australia](https://mlsanity.com/paper/2007.02325/) (2020)
- [DepressionNet: A Novel Summarization Boosted Deep Framework for Depression Detection on Social Media](https://mlsanity.com/paper/2105.10878/) (2021)
- [User-Aware Multilingual Abusive Content Detection in Social Media](https://mlsanity.com/paper/2410.21321/) (2024)
- [COVID-19 Twitter Dataset with Latent Topics, Sentiments and Emotions Attributes](https://mlsanity.com/paper/2007.06954/) (2020)
- [Dynamic Graph Representation Learning for Depression Screening with Transformer](https://mlsanity.com/paper/2305.06447/) (2023)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-aware Semi-supervised Ensemble Teacher Framework for Multilingual Depression Detection](https://mlsanity.com/paper/2512.24772/) (2025)

## Abstract (arXiv)

The recent coronavirus disease (Covid-19) has become a pandemic and has affected the entire globe. During the pandemic, we have observed a spike in cases related to mental health, such as anxiety, stress, and depression. Depression significantly influences most diseases worldwide, making it difficult to detect mental health conditions in people due to unawareness and unwillingness to consult a doctor. However, nowadays, people extensively use online social media platforms to express their emotions and thoughts. Hence, social media platforms are now becoming a large data source that can be utilized for detecting depression and mental illness. However, existing approaches often overlook data sparsity in tweets and the multimodal aspects of social media. In this paper, we propose a novel multimodal framework that combines textual, user-specific, and image analysis to detect depression among social media users. To provide enough context about the user's emotional state, we propose (i) an extrinsic feature by harnessing the URLs present in tweets and (ii) extracting textual content present in images posted in tweets. We also extract five sets of features belonging to different modalities to describe a user. Additionally, we introduce a Deep Learning model, the Visual Neural Network (VNN), to generate embeddings of user-posted images, which are used to create the visual feature vector for prediction. We contribute a curated Covid-19 dataset of depressed and non-depressed users for research purposes and demonstrate the effectiveness of our model in detecting depression during the Covid-19 outbreak. Our model outperforms existing state-of-the-art methods over a benchmark dataset by 2%-8% and produces promising results on the Covid-19 dataset. Our analysis highlights the impact of each modality and provides valuable insights into users' mental and emotional states.