# DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text

arXiv: https://arxiv.org/abs/2209.05286 · HTML: https://mlsanity.com/paper/2209.05286/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 62 references that are in the map.
- First posted on arXiv 12 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment](https://mlsanity.com/paper/1804.08117/) (2018)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [Did You Really Just Have a Heart Attack? Towards Robust Detection of Personal Health Mentions in Social Media](https://mlsanity.com/paper/1802.09130/) (2018)
- [What Twitter Profile and Posted Images Reveal About Depression and Anxiety](https://mlsanity.com/paper/1904.02670/) (2019)
- [Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Health](https://mlsanity.com/paper/2109.13770/) (2021)
- [Detecting cognitive impairments by agreeing on interpretations of linguistic features](https://mlsanity.com/paper/1808.06570/) (2018)
- [Robustness and Sensitivity of BERT Models Predicting Alzheimer's Disease from Text](https://mlsanity.com/paper/2109.11888/) (2021)

## Built on by (most cited first, 2 of 2)

- [WundtGPT: Shaping Large Language Models To Be An Empathetic, Proactive Psychologist](https://mlsanity.com/paper/2406.15474/) (2024)
- [Cost-effective Models for Detecting Depression from Speech](https://mlsanity.com/paper/2302.09214/) (2023)

## Abstract (arXiv)

Models that accurately detect depression from text are important tools for addressing the post-pandemic mental health crisis. BERT-based classifiers' promising performance and the off-the-shelf availability make them great candidates for this task. However, these models are known to suffer from performance inconsistencies and poor generalization. In this paper, we introduce the DECK (DEpression ChecKlist), depression-specific model behavioural tests that allow better interpretability and improve generalizability of BERT classifiers in depression domain. We create 23 tests to evaluate BERT, RoBERTa and ALBERT depression classifiers on three datasets, two Twitter-based and one clinical interview-based. Our evaluation shows that these models: 1) are robust to certain gender-sensitive variations in text; 2) rely on the important depressive language marker of the increased use of first person pronouns; 3) fail to detect some other depression symptoms like suicidal ideation. We also demonstrate that DECK tests can be used to incorporate symptom-specific information in the training data and consistently improve generalizability of all three BERT models, with an out-of-distribution F1-score increase of up to 53.93%.