# An End-to-End Set Transformer for User-Level Classification of Depression and Gambling Disorder

arXiv: https://arxiv.org/abs/2207.00753 · HTML: https://mlsanity.com/paper/2207.00753/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 2 Jul 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Early Risk Detection of Pathological Gambling, Self-Harm and Depression Using BERT](https://mlsanity.com/paper/2106.16175/) (2021)
- [Detecting Early Onset of Depression from Social Media Text using Learned Confidence Scores](https://mlsanity.com/paper/2011.01695/) (2020)
- [A Psychologically Informed Part-of-Speech Analysis of Depression in Social Media](https://mlsanity.com/paper/2108.00279/) (2021)

## Built on by (most cited first, 4 of 4)

- [On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlook](https://mlsanity.com/paper/2410.08793/) (2024)
- [Hierarchical attention interpretation: an interpretable speech-level transformer for bi-modal depression detection](https://mlsanity.com/paper/2309.13476/) (2023)
- [Automatic detection of problem-gambling signs from online texts using large language models](https://mlsanity.com/paper/2312.00804/) (2023)
- [Risk prediction of pathological gambling on social media](https://mlsanity.com/paper/2403.19358/) (2024)

## Abstract (arXiv)

This work proposes a transformer architecture for user-level classification of gambling addiction and depression that is trainable end-to-end. As opposed to other methods that operate at the post level, we process a set of social media posts from a particular individual, to make use of the interactions between posts and eliminate label noise at the post level. We exploit the fact that, by not injecting positional encodings, multi-head attention is permutation invariant and we process randomly sampled sets of texts from a user after being encoded with a modern pretrained sentence encoder (RoBERTa / MiniLM). Moreover, our architecture is interpretable with modern feature attribution methods and allows for automatic dataset creation by identifying discriminating posts in a user's text-set. We perform ablation studies on hyper-parameters and evaluate our method for the eRisk 2022 Lab on early detection of signs of pathological gambling and early risk detection of depression. The method proposed by our team BLUE obtained the best ERDE5 score of 0.015, and the second-best ERDE50 score of 0.009 for pathological gambling detection. For the early detection of depression, we obtained the second-best ERDE50 of 0.027.