# Enhancing Speech Emotion Recognition through Segmental Average Pooling of Self-Supervised Learning Features

arXiv: https://arxiv.org/abs/2410.12416 · HTML: https://mlsanity.com/paper/2410.12416/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Speech Emotion Recognition Using Multi-hop Attention Mechanism](https://mlsanity.com/paper/1904.10788/) (2019)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Attention Driven Fusion for Multi-Modal Emotion Recognition](https://mlsanity.com/paper/2009.10991/) (2020)
- [Improving Speech Emotion Recognition Through Focus and Calibration Attention Mechanisms](https://mlsanity.com/paper/2208.10491/) (2022)
- [Supervised Contrastive Learning with Nearest Neighbor Search for Speech Emotion Recognition](https://mlsanity.com/paper/2308.16485/) (2023)

## Abstract (arXiv)

Speech Emotion Recognition (SER) analyzes human emotions expressed through speech. Self-supervised learning (SSL) offers a promising approach to SER by learning meaningful representations from a large amount of unlabeled audio data. However, existing SSL-based methods rely on Global Average Pooling (GAP) to represent audio signals, treating speech and non-speech segments equally. This can lead to dilution of informative speech features by irrelevant non-speech information. To address this, the paper proposes Segmental Average Pooling (SAP), which selectively focuses on informative speech segments while ignoring non-speech segments. By applying both GAP and SAP to SSL features, our approach utilizes overall speech signal information from GAP and specific information from SAP, leading to improved SER performance. Experiments show state-of-the-art results on the IEMOCAP for English and superior performance on KEMDy19 for Korean datasets in both unweighted and weighted accuracies.