# Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech

arXiv: https://arxiv.org/abs/2510.08593 · HTML: https://mlsanity.com/paper/2510.08593/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysis](https://mlsanity.com/paper/2002.09283/) (2020)
- [Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Model](https://mlsanity.com/paper/2202.08210/) (2022)
- [SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech](https://mlsanity.com/paper/2203.03812/) (2022)
- [MFCC-based Recurrent Neural Network for Automatic Clinical Depression Recognition and Assessment from Speech](https://mlsanity.com/paper/1909.07208/) (2019)
- [Self-supervised representations in speech-based depression detection](https://mlsanity.com/paper/2305.12263/) (2023)
- [A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models](https://mlsanity.com/paper/2408.13678/) (2024)
- [DEPA: Self-Supervised Audio Embedding for Depression Detection](https://mlsanity.com/paper/1910.13028/) (2019)
- [AudVowelConsNet: A Phoneme-Level Based Deep CNN Architecture for Clinical Depression Diagnosis](https://mlsanity.com/paper/2010.16201/) (2020)
- [What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis](https://mlsanity.com/paper/2401.17632/) (2024)
- [Automated speech-based screening of depression using deep convolutional neural networks](https://mlsanity.com/paper/1912.01115/) (2019)
- [A Topic-Attentive Transformer-based Model For Multimodal Depression Detection](https://mlsanity.com/paper/2206.13256/) (2022)
- [Automatic Detection of Depression in Speech Using Ensemble Convolutional Neural Networks](https://mlsanity.com/paper/2402.02830/) (2024)
- [A Step Towards Preserving Speakers' Identity While Detecting Depression Via Speaker Disentanglement](https://mlsanity.com/paper/2206.09530/) (2022)
- [Automatic Depression Detection via Learning and Fusing Features from Visual Cues](https://mlsanity.com/paper/2203.00304/) (2022)
- [Exploring ASR-Based Wav2Vec2 for Automated Speech Disorder Assessment: Insights and Analysis](https://mlsanity.com/paper/2410.08250/) (2024)
- [Context-Aware Deep Learning for Multi Modal Depression Detection](https://mlsanity.com/paper/2412.19209/) (2024)
- [Cost-effective Models for Detecting Depression from Speech](https://mlsanity.com/paper/2302.09214/) (2023)

## Built on by (most cited first, 2 of 2)

- [DepFlow: Disentangled Speech Generation to Mitigate Semantic Bias in Depression Detection](https://mlsanity.com/paper/2601.00303/) (2026)
- [Can Conversational Temporal Dynamics Improve Depression Detection in Dyads? A Preliminary Investigation in Multi-Modality Perspectives](https://mlsanity.com/paper/2607.03744/) (2026)

## Abstract (arXiv)

Speech-based depression detection (SDD) has emerged as a non-invasive and scalable alternative to conventional clinical assessments. However, existing methods still struggle to capture robust depression-related speech characteristics, which are sparse and heterogeneous. Although pretrained self-supervised learning (SSL) models provide rich representations, most recent SDD studies extract features from a single layer of the pretrained SSL model for the downstream classifier. This practice overlooks the complementary roles of low-level acoustic features and high-level semantic information inherently encoded in different SSL model layers. To explicitly model interactions between acoustic and semantic representations within an utterance, we propose a hierarchical adaptive representation encoder with prior knowledge that disengages and re-aligns acoustic and semantic information through asymmetric cross-attention, enabling fine-grained acoustic patterns to be interpreted in semantic context. In addition, a Connectionist Temporal Classification (CTC) objective is applied as auxiliary supervision to handle the irregular temporal distribution of depressive characteristics without requiring frame-level annotations. Experiments on DAIC-WOZ and MODMA demonstrate that HAREN-CTC consistently outperforms existing methods under both performance upper-bound evaluation and generalization evaluation settings, achieving Macro F1 scores of 0.81 and 0.82 respectively in upper-bound evaluation, and maintaining superior performance with statistically significant improvements in precision and AUC under rigorous cross-validation. These findings suggest that modeling hierarchical acoustic-semantic interactions better reflects how depressive characteristics manifest in natural speech, enabling scalable and objective depression assessment.