# Speech Swin-Transformer: Exploring a Hierarchical Transformer with Shifted Windows for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2401.10536 · HTML: https://mlsanity.com/paper/2401.10536/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 19 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Hybrid Curriculum Learning for Emotion Recognition in Conversation](https://mlsanity.com/paper/2112.11718/) (2021)
- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Speech Emotion Recognition via an Attentive Time-Frequency Neural Network](https://mlsanity.com/paper/2210.12430/) (2022)
- [Learning Local to Global Feature Aggregation for Speech Emotion Recognition](https://mlsanity.com/paper/2306.01491/) (2023)
- [Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method for Speech Emotion Recognition](https://mlsanity.com/paper/2308.14568/) (2023)

## Built on by (most cited first, 4 of 4)

- [Swin-BERT: A Feature Fusion System designed for Speech-based Alzheimer's Dementia Detection](https://mlsanity.com/paper/2410.07277/) (2024)
- [Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematics](https://mlsanity.com/paper/2511.07955/) (2025)
- [AVATAAR: Agentic Video Answering via Temporal Adaptive Alignment and Reasoning](https://mlsanity.com/paper/2511.15578/) (2025)
- [mmWave-Diffusion:A Novel Framework for Respiration Sensing Using Observation-Anchored Conditional Diffusion Model](https://mlsanity.com/paper/2603.20700/) (2026)

## Abstract (arXiv)

Swin-Transformer has demonstrated remarkable success in computer vision by leveraging its hierarchical feature representation based on Transformer. In speech signals, emotional information is distributed across different scales of speech features, e.,g., word, phrase, and utterance. Drawing above inspiration, this paper presents a hierarchical speech Transformer with shifted windows to aggregate multi-scale emotion features for speech emotion recognition (SER), called Speech Swin-Transformer. Specifically, we first divide the speech spectrogram into segment-level patches in the time domain, composed of multiple frame patches. These segment-level patches are then encoded using a stack of Swin blocks, in which a local window Transformer is utilized to explore local inter-frame emotional information across frame patches of each segment patch. After that, we also design a shifted window Transformer to compensate for patch correlations near the boundaries of segment patches. Finally, we employ a patch merging operation to aggregate segment-level emotional features for hierarchical speech representation by expanding the receptive field of Transformer from frame-level to segment-level. Experimental results demonstrate that our proposed Speech Swin-Transformer outperforms the state-of-the-art methods.