# Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detection

arXiv: https://arxiv.org/abs/2412.18614 · HTML: https://mlsanity.com/paper/2412.18614/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 70 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition](https://mlsanity.com/paper/2110.06309/) (2021)
- [Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Model](https://mlsanity.com/paper/2202.08210/) (2022)
- [It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers](https://mlsanity.com/paper/2301.05453/) (2023)
- [Self-supervised representations in speech-based depression detection](https://mlsanity.com/paper/2305.12263/) (2023)
- [Speaker Normalization for Self-supervised Speech Emotion Recognition](https://mlsanity.com/paper/2202.01252/) (2022)
- [Climate and Weather: Inspecting Depression Detection via Emotion Recognition](https://mlsanity.com/paper/2204.14099/) (2022)
- [A knowledge-driven vowel-based approach of depression classification from speech using data augmentation](https://mlsanity.com/paper/2210.15261/) (2022)

## Built on by (most cited first, 1 of 1)

- [MA-DLE: Speech-based Automatic Depression Level Estimation via Memory Augmentation](https://mlsanity.com/paper/2606.11197/) (2026)

## Abstract (arXiv)

Previous studies have demonstrated that emotional features from a single acoustic sentiment label can enhance depression diagnosis accuracy. Additionally, according to the Emotion Context-Insensitivity theory and our pilot study, individuals with depression might convey negative emotional content in an unexpectedly calm manner, showing a high degree of inconsistency in emotional expressions during natural conversations. So far, few studies have recognized and leveraged the emotional expression inconsistency for depression detection. In this paper, a multimodal cross-attention method is presented to capture the Acoustic-Textual Emotional Inconsistency (ATEI) information. This is achieved by analyzing the intricate local and long-term dependencies of emotional expressions across acoustic and textual domains, as well as the mismatch between the emotional content within both domains. A Transformer-based model is then proposed to integrate this ATEI information with various fusion strategies for detecting depression. Furthermore, a scaling technique is employed to adjust the ATEI feature degree during the fusion process, thereby enhancing the model's ability to discern patients with depression across varying levels of severity. To best of our knowledge, this work is the first to incorporate emotional expression inconsistency information into depression detection. Experimental results on a counseling conversational dataset illustrate the effectiveness of our method.