# It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers

arXiv: https://arxiv.org/abs/2301.05453 · HTML: https://mlsanity.com/paper/2301.05453/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 60 references that are in the map.
- First posted on arXiv 13 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://mlsanity.com/paper/1709.01848/) (2017)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences](https://mlsanity.com/paper/1804.07000/) (2018)
- [Instagram photos reveal predictive markers of depression](https://mlsanity.com/paper/1608.03282/) (2016)
- [What Twitter Profile and Posted Images Reveal About Depression and Anxiety](https://mlsanity.com/paper/1904.02670/) (2019)
- [Identifying Moments of Change from Longitudinal User Text](https://mlsanity.com/paper/2205.05593/) (2022)
- [Early Risk Detection of Pathological Gambling, Self-Harm and Depression Using BERT](https://mlsanity.com/paper/2106.16175/) (2021)
- [See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Data](https://mlsanity.com/paper/1912.01131/) (2019)
- [Deep Bag-of-Sub-Emotions for Depression Detection in Social Media](https://mlsanity.com/paper/2103.01334/) (2021)
- [Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses](https://mlsanity.com/paper/2206.11155/) (2022)

## Built on by (most cited first, 18 of 18)

- [Read, Diagnose and Chat: Towards Explainable and Interactive LLMs-Augmented Depression Detection in Social Media](https://mlsanity.com/paper/2305.05138/) (2023)
- [Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues](https://mlsanity.com/paper/2401.02746/) (2024)
- [Explainable Depression Symptom Detection in Social Media](https://mlsanity.com/paper/2310.13664/) (2023)
- [Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges](https://mlsanity.com/paper/2407.16804/) (2024)
- [On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlook](https://mlsanity.com/paper/2410.08793/) (2024)
- [Utilizing ChatGPT Generated Data to Retrieve Depression Symptoms from Social Media](https://mlsanity.com/paper/2307.02313/) (2023)
- [A Survey on Multilingual Mental Disorders Detection from Social Media Data](https://mlsanity.com/paper/2505.15556/) (2025)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking](https://mlsanity.com/paper/2403.09717/) (2024)
- [3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detection](https://mlsanity.com/paper/2407.09020/) (2024)
- [Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detection](https://mlsanity.com/paper/2412.18614/) (2024)
- [A Multimodal Framework for Depression Detection during Covid-19 via Harvesting Social Media: A Novel Dataset and Method](https://mlsanity.com/paper/2511.00424/) (2025)
- [Risk prediction of pathological gambling on social media](https://mlsanity.com/paper/2403.19358/) (2024)
- [Learning Longitudinal Stress Dynamics from Irregular Self-Reports via Time Embeddings](https://mlsanity.com/paper/2509.01569/) (2025)
- [MoME: Estimating Psychological Traits from Gait with Multi-Stage Mixture of Movement Experts](https://mlsanity.com/paper/2510.04654/) (2025)
- [Tri-Modal Severity Fused Diagnosis across Depression and Post-traumatic Stress Disorders](https://mlsanity.com/paper/2510.20239/) (2025)
- [IDRL: An Individual-Aware Multimodal Depression-Related Representation Learning Framework for Depression Diagnosis](https://mlsanity.com/paper/2603.11644/) (2026)
- [VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection](https://mlsanity.com/paper/2609.05537/) (2026)

## Abstract (arXiv)

Depression detection from user-generated content on the internet has been a long-lasting topic of interest in the research community, providing valuable screening tools for psychologists. The ubiquitous use of social media platforms lays out the perfect avenue for exploring mental health manifestations in posts and interactions with other users. Current methods for depression detection from social media mainly focus on text processing, and only a few also utilize images posted by users. In this work, we propose a flexible time-enriched multimodal transformer architecture for detecting depression from social media posts, using pretrained models for extracting image and text embeddings. Our model operates directly at the user-level, and we enrich it with the relative time between posts by using time2vec positional embeddings. Moreover, we propose another model variant, which can operate on randomly sampled and unordered sets of posts to be more robust to dataset noise. We show that our method, using EmoBERTa and CLIP embeddings, surpasses other methods on two multimodal datasets, obtaining state-of-the-art results of 0.931 F1 score on a popular multimodal Twitter dataset, and 0.902 F1 score on the only multimodal Reddit dataset.