# Neural Representations for Modeling Variation in Speech

arXiv: https://arxiv.org/abs/2011.12649 · HTML: https://mlsanity.com/paper/2011.12649/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 14 of 14)

- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [Encoding of lexical tone in self-supervised models of spoken language](https://mlsanity.com/paper/2403.16865/) (2024)
- [Quantifying Language Variation Acoustically with Few Resources](https://mlsanity.com/paper/2205.02694/) (2022)
- [Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens](https://mlsanity.com/paper/2402.02302/) (2024)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [Self-supervised Speech Representations Still Struggle with African American Vernacular English](https://mlsanity.com/paper/2408.14262/) (2024)
- [Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations](https://mlsanity.com/paper/2604.02102/) (2026)
- [Self-supervised Speech Comparison for L2 Phone, Rhythm, and Intonation Scoring](https://mlsanity.com/paper/2607.13721/) (2026)
- [Low-resource Accent Classification in Geographically-proximate Settings: A Forensic and Sociophonetics Perspective](https://mlsanity.com/paper/2206.12759/) (2022)
- [Supervised Acoustic Embeddings And Their Transferability Across Languages](https://mlsanity.com/paper/2301.01020/) (2023)
- [From `Snippet-lects' to Doculects and Dialects: Leveraging Neural Representations of Speech for Placing Audio Signals in a Language Landscape](https://mlsanity.com/paper/2305.18602/) (2023)
- [Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis](https://mlsanity.com/paper/2503.04814/) (2025)
- [Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception](https://mlsanity.com/paper/2506.17542/) (2025)
- [Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition](https://mlsanity.com/paper/2601.06972/) (2026)

## Abstract (arXiv)

Variation in speech is often quantified by comparing phonetic transcriptions of the same utterance. However, manually transcribing speech is time-consuming and error prone. As an alternative, therefore, we investigate the extraction of acoustic embeddings from several self-supervised neural models. We use these representations to compute word-based pronunciation differences between non-native and native speakers of English, and between Norwegian dialect speakers. For comparison with several earlier studies, we evaluate how well these differences match human perception by comparing them with available human judgements of similarity. We show that speech representations extracted from a specific type of neural model (i.e. Transformers) lead to a better match with human perception than two earlier approaches on the basis of phonetic transcriptions and MFCC-based acoustic features. We furthermore find that features from the neural models can generally best be extracted from one of the middle hidden layers than from the final layer. We also demonstrate that neural speech representations not only capture segmental differences, but also intonational and durational differences that cannot adequately be represented by a set of discrete symbols used in phonetic transcriptions.