# Analyzing analytical methods: The case of phonology in neural models of spoken language

arXiv: https://arxiv.org/abs/2004.07070 · HTML: https://mlsanity.com/paper/2004.07070/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model](https://mlsanity.com/paper/2306.05720/) (2023)
- [Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0](https://mlsanity.com/paper/2407.03005/) (2024)
- [Wave to Syntax: Probing spoken language models for syntax](https://mlsanity.com/paper/2305.18957/) (2023)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings](https://mlsanity.com/paper/2109.10179/) (2021)
- [Discrete representations in neural models of spoken language](https://mlsanity.com/paper/2105.05582/) (2021)
- [A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speech](https://mlsanity.com/paper/2405.08237/) (2024)
- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)
- [Using Shapley interactions to understand how models use structure](https://mlsanity.com/paper/2403.13106/) (2024)
- [Prompting Underestimates LLM Capability for Time Series Classification](https://mlsanity.com/paper/2601.03464/) (2026)
- [Tracking the emergence of linguistic structure in self-supervised models learning from speech](https://mlsanity.com/paper/2604.02043/) (2026)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)
- [Designing and Contextualising Probes for African Languages](https://mlsanity.com/paper/2505.10081/) (2025)

## Abstract (arXiv)

Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.