# Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs

arXiv: https://arxiv.org/abs/2403.17299 · HTML: https://mlsanity.com/paper/2403.17299/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Do self-supervised speech and language models extract similar representations as human brain?](https://mlsanity.com/paper/2310.04645/) (2023)

## Built on by (most cited first, 24 of 24)

- [LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encoder](https://mlsanity.com/paper/2502.20344/) (2025)
- [XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs](https://mlsanity.com/paper/2502.19737/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models](https://mlsanity.com/paper/2506.02132/) (2025)
- [HARP: Hallucination Detection via Reasoning Subspace Projection](https://mlsanity.com/paper/2509.11536/) (2025)
- [Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations](https://mlsanity.com/paper/2509.15655/) (2025)
- [Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence](https://mlsanity.com/paper/2411.07533/) (2024)
- [LLM Hallucination Detection: A Fast Fourier Transform Method Based on Hidden Layer Temporal Signals](https://mlsanity.com/paper/2509.13154/) (2025)
- [CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment](https://mlsanity.com/paper/2502.11066/) (2025)
- [SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Model](https://mlsanity.com/paper/2502.19960/) (2025)
- [Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations](https://mlsanity.com/paper/2601.03779/) (2026)
- [Bi-directional Model Cascading with Proxy Confidence](https://mlsanity.com/paper/2504.19391/) (2025)
- [Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect](https://mlsanity.com/paper/2509.12065/) (2025)
- [From Chains to DAGs: Probing the Graph Structure of Reasoning in LLMs](https://mlsanity.com/paper/2601.17593/) (2026)
- [Dual Alignment Between Language Model Layers and Human Sentence Processing](https://mlsanity.com/paper/2604.18563/) (2026)
- [Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models](https://mlsanity.com/paper/2409.12435/) (2024)
- [LLM Hallucination Detection: HSAD](https://mlsanity.com/paper/2509.23580/) (2025)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust](https://mlsanity.com/paper/2606.07822/) (2026)
- [Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains](https://mlsanity.com/paper/2603.14400/) (2026)
- [When Models Know More Than They Say: Probing Analogical Reasoning in LLMs](https://mlsanity.com/paper/2604.03877/) (2026)
- [MADS: Model-Aware Diverse Core Set Selection for Instruction Tuning](https://mlsanity.com/paper/2605.30857/) (2026)
- [ProbeScale: Probing Analysis to Optimize Neural Scaling Laws for Efficient Small Language Model Inference](https://mlsanity.com/paper/2606.01806/) (2026)
- [Grammatical "grandmother neurons" are rare in LLMs](https://mlsanity.com/paper/2609.29328/) (2026)

## Abstract (arXiv)

Inspired by cognitive neuroscience studies, we introduce a novel `decoding probing' method that uses minimal pairs benchmark (BLiMP) to probe internal linguistic characteristics in neural language models layer by layer. By treating the language model as the `brain' and its representations as `neural activations', we decode grammaticality labels of minimal pairs from the intermediate layers' representations. This approach reveals: 1) Self-supervised language models capture abstract linguistic structures in intermediate layers that GloVe and RNN language models cannot learn. 2) Information about syntactic grammaticality is robustly captured through the first third layers of GPT-2 and also distributed in later layers. As sentence complexity increases, more layers are required for learning grammatical capabilities. 3) Morphological and semantics/syntax interface-related features are harder to capture than syntax. 4) For Transformer-based models, both embeddings and attentions capture grammatical features but show distinct patterns. Different attention heads exhibit similar tendencies toward various linguistic phenomena, but with varied contributions.