# Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations

arXiv: https://arxiv.org/abs/2509.15655 · HTML: https://mlsanity.com/paper/2509.15655/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 30 of 44)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Self-Supervised Speech Representations are More Phonetic than Semantic](https://mlsanity.com/paper/2406.08619/) (2024)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [Towards audio language modeling -- an overview](https://mlsanity.com/paper/2402.13236/) (2024)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [Probing phoneme, language and speaker information in unsupervised speech representations](https://mlsanity.com/paper/2203.16193/) (2022)
- [Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems](https://mlsanity.com/paper/1709.04482/) (2017)
- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [Similarity Analysis of Self-Supervised Speech Representations](https://mlsanity.com/paper/2010.11481/) (2020)

## Built on by (most cited first, 5 of 5)

- [Abstraction Induces the Brain Alignment of Language and Speech Models](https://mlsanity.com/paper/2602.04081/) (2026)
- [Resurfacing Paralinguistic Awareness in Large Audio Language Models](https://mlsanity.com/paper/2603.11947/) (2026)
- [Tracking the emergence of linguistic structure in self-supervised models learning from speech](https://mlsanity.com/paper/2604.02043/) (2026)
- [Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification](https://mlsanity.com/paper/2606.07479/) (2026)
- [Grammatical "grandmother neurons" are rare in LLMs](https://mlsanity.com/paper/2609.29328/) (2026)

## Abstract (arXiv)

Transformer-based speech language models (SLMs) have significantly improved neural speech recognition and understanding. While existing research has examined how well SLMs encode shallow acoustic and phonetic features, the extent to which SLMs encode nuanced syntactic and conceptual features remains unclear. By drawing parallels with linguistic competence assessments for large language models, this study is the first to systematically evaluate the presence of contextual syntactic and semantic features across SLMs for self-supervised learning (S3M), automatic speech recognition (ASR), speech compression (codec), and as the encoder for auditory large language models (AudioLLMs). Through minimal pair designs and diagnostic feature analysis across 71 tasks spanning diverse linguistic levels, our layer-wise and time-resolved analysis uncovers that 1) all speech encode grammatical features more robustly than conceptual ones.