# Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2402.02617 · HTML: https://mlsanity.com/paper/2402.02617/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples](https://mlsanity.com/paper/1912.04853/) (2019)
- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models](https://mlsanity.com/paper/2210.16043/) (2022)
- [A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings](https://mlsanity.com/paper/2012.02221/) (2020)

## Built on by (most cited first, 8 of 8)

- [Crossmodal ASR Error Correction with Discrete Speech Units](https://mlsanity.com/paper/2405.16677/) (2024)
- [SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection](https://mlsanity.com/paper/2407.18517/) (2024)
- [Learn from Real: Reality Defender's Submission to ASVspoof5 Challenge](https://mlsanity.com/paper/2410.07379/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models](https://mlsanity.com/paper/2409.16920/) (2024)
- [Addressing Emotion Bias in Music Emotion Recognition and Generation with Frechet Audio Distance](https://mlsanity.com/paper/2409.15545/) (2024)
- [Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations](https://mlsanity.com/paper/2409.17899/) (2024)
- [Color-based Emotion Representation for Speech Emotion Recognition](https://mlsanity.com/paper/2602.16256/) (2026)

## Abstract (arXiv)

The efficacy of self-supervised speech models has been validated, yet the optimal utilization of their representations remains challenging across diverse tasks. In this study, we delve into Acoustic Word Embeddings (AWEs), a fixed-length feature derived from continuous representations, to explore their advantages in specific tasks. AWEs have previously shown utility in capturing acoustic discriminability. In light of this, we propose measuring layer-wise similarity between AWEs and word embeddings, aiming to further investigate the inherent context within AWEs. Moreover, we evaluate the contribution of AWEs, in comparison to other types of speech features, in the context of Speech Emotion Recognition (SER). Through a comparative experiment and a layer-wise accuracy analysis on two distinct corpora, IEMOCAP and ESD, we explore differences between AWEs and raw self-supervised representations, as well as the proper utilization of AWEs alone and in combination with word embeddings. Our findings underscore the acoustic context conveyed by AWEs and showcase the highly competitive SER accuracies by appropriately employing AWEs.