# Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Features

arXiv: https://arxiv.org/abs/2409.09511 · HTML: https://mlsanity.com/paper/2409.09511/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 10 of 10)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Probing Speech Emotion Recognition Transformers for Linguistic Knowledge](https://mlsanity.com/paper/2204.00400/) (2022)
- [Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition](https://mlsanity.com/paper/2308.08713/) (2023)

## Built on by (most cited first, 2 of 2)

- [Acoustic and Machine Learning Methods for Speech-Based Suicide Risk Assessment: A Systematic Review](https://mlsanity.com/paper/2505.18195/) (2025)
- [Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception](https://mlsanity.com/paper/2506.17542/) (2025)

## Abstract (arXiv)

Pre-trained deep learning embeddings have consistently shown superior performance over handcrafted acoustic features in speech emotion recognition (SER). However, unlike acoustic features with clear physical meaning, these embeddings lack clear interpretability. Explaining these embeddings is crucial for building trust in healthcare and security applications and advancing the scientific understanding of the acoustic information that is encoded in them. This paper proposes a modified probing approach to explain deep learning embeddings in the SER space. We predict interpretable acoustic features (e.g., f0, loudness) from (i) the complete set of embeddings and (ii) a subset of the embedding dimensions identified as most important for predicting each emotion. If the subset of the most important dimensions better predicts a given emotion than all dimensions and also predicts specific acoustic features more accurately, we infer those acoustic features are important for the embedding model for the given task. We conducted experiments using the WavLM embeddings and eGeMAPS acoustic features as audio representations, applying our method to the RAVDESS and SAVEE emotional speech datasets. Based on this evaluation, we demonstrate that Energy, Frequency, Spectral, and Temporal categories of acoustic features provide diminishing information to SER in that order, demonstrating the utility of the probing classifier method to relate embeddings to interpretable acoustic features.