# SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech

arXiv: https://arxiv.org/abs/2203.03812 · HTML: https://mlsanity.com/paper/2203.03812/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2022.

## Built on by (most cited first, 20 of 20)

- [CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting](https://mlsanity.com/paper/2305.12095/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2310.05804/) (2023)
- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [HiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection](https://mlsanity.com/paper/2408.03648/) (2024)
- [CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation](https://mlsanity.com/paper/2310.11295/) (2023)
- [MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction](https://mlsanity.com/paper/2401.13260/) (2024)
- [DWFormer: Dynamic Window transFormer for Speech Emotion Recognition](https://mlsanity.com/paper/2303.01694/) (2023)
- [A knowledge-driven vowel-based approach of depression classification from speech using data augmentation](https://mlsanity.com/paper/2210.15261/) (2022)
- [A Change of Heart: Improving Speech Emotion Recognition through Speech-to-Text Modality Conversion](https://mlsanity.com/paper/2307.11584/) (2023)
- [Toward Knowledge-Driven Speech-Based Models of Depression: Leveraging Spectrotemporal Variations in Speech Vowels](https://mlsanity.com/paper/2210.02527/) (2022)
- [Robust and Explainable Depression Identification from Speech Using Vowel-Based Ensemble Learning Approaches](https://mlsanity.com/paper/2410.18298/) (2024)
- [Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech](https://mlsanity.com/paper/2510.08593/) (2025)
- [It Hears, It Sees too: Multi-Modal LLM for Depression Detection By Integrating Visual Understanding into Audio Language Models](https://mlsanity.com/paper/2511.19877/) (2025)
- [FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency](https://mlsanity.com/paper/2410.03719/) (2024)
- [Generalizable Audio Spoofing Detection using Non-Semantic Representations](https://mlsanity.com/paper/2509.00186/) (2025)
- [On the Contribution of Lexical Features to Speech Emotion Recognition](https://mlsanity.com/paper/2509.05634/) (2025)
- [DepFlow: Disentangled Speech Generation to Mitigate Semantic Bias in Depression Detection](https://mlsanity.com/paper/2601.00303/) (2026)

## Abstract (arXiv)

Transformer has obtained promising results on cognitive speech signal processing field, which is of interest in various applications ranging from emotion to neurocognitive disorder analysis. However, most works treat speech signal as a whole, leading to the neglect of the pronunciation structure that is unique to speech and reflects the cognitive process. Meanwhile, Transformer has heavy computational burden due to its full attention operation. In this paper, a hierarchical efficient framework, called SpeechFormer, which considers the structural characteristics of speech, is proposed and can be served as a general-purpose backbone for cognitive speech signal processing. The proposed SpeechFormer consists of frame, phoneme, word and utterance stages in succession, each performing a neighboring attention according to the structural pattern of speech with high computational efficiency. SpeechFormer is evaluated on speech emotion recognition (IEMOCAP & MELD) and neurocognitive disorder detection (Pitt & DAIC-WOZ) tasks, and the results show that SpeechFormer outperforms the standard Transformer-based framework while greatly reducing the computational cost. Furthermore, our SpeechFormer achieves comparable results to the state-of-the-art approaches.