# Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spotting

arXiv: https://arxiv.org/abs/2506.11096 · HTML: https://mlsanity.com/paper/2506.11096/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 11 of 11)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning](https://mlsanity.com/paper/2206.13378/) (2022)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)

## Built on by (most cited first, 1 of 1)

- [The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation](https://mlsanity.com/paper/2604.26347/) (2026)

## Abstract (arXiv)

Pretrained speech representations like wav2vec2 and HuBERT exhibit strong anisotropy, leading to high similarity between random embeddings. While widely observed, the impact of this property on downstream tasks remains unclear. This work evaluates anisotropy in keyword spotting for computational documentary linguistics. Using Dynamic Time Warping, we show that despite anisotropy, wav2vec2 similarity measures effectively identify words without transcription. Our results highlight the robustness of these representations, which capture phonetic structures and generalize across speakers. Our results underscore the importance of pretraining in learning rich and invariant speech representations.