# A Tale of a Probe and a Parser

arXiv: https://arxiv.org/abs/2005.01641 · HTML: https://mlsanity.com/paper/2005.01641/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 30 of 40)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [Do Transformers Parse while Predicting the Masked Word?](https://mlsanity.com/paper/2303.08117/) (2023)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [A Non-Linear Structural Probe](https://mlsanity.com/paper/2105.10185/) (2021)
- [Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing](https://mlsanity.com/paper/2106.02559/) (2021)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT](https://mlsanity.com/paper/2510.00296/) (2025)
- [AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models](https://mlsanity.com/paper/2206.11719/) (2022)
- [Probing for Constituency Structure in Neural Language Models](https://mlsanity.com/paper/2204.06201/) (2022)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)
- [When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes](https://mlsanity.com/paper/2204.09722/) (2022)
- [Probing for Labeled Dependency Trees](https://mlsanity.com/paper/2203.12971/) (2022)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions](https://mlsanity.com/paper/2202.09166/) (2022)
- [Latent Topology Induction for Understanding Contextualized Representations](https://mlsanity.com/paper/2206.01512/) (2022)
- [How Well Do Large Language Models Understand Syntax? An Evaluation by Asking Natural Language Questions](https://mlsanity.com/paper/2311.08287/) (2023)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Schr"odinger's Tree -- On Syntax and Neural Language Models](https://mlsanity.com/paper/2110.08887/) (2021)
- [A Latent-Variable Model for Intrinsic Probing](https://mlsanity.com/paper/2201.08214/) (2022)
- [The Architectural Bottleneck Principle](https://mlsanity.com/paper/2211.06420/) (2022)
- [Probing the Category of Verbal Aspect in Transformer Language Models](https://mlsanity.com/paper/2406.02335/) (2024)
- [Visualizing and Explaining Language Models](https://mlsanity.com/paper/2205.10238/) (2022)
- [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024)
- [Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models](https://mlsanity.com/paper/2412.05353/) (2024)
- [Automatic Expansion of Domain-Specific Affective Models for Web Intelligence Applications](https://mlsanity.com/paper/2102.00827/) (2021)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)

## Abstract (arXiv)

Measuring what linguistic information is encoded in neural models of language has become popular in NLP. Researchers approach this enterprise by training "probes" - supervised models designed to extract linguistic structure from another model's output. One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations. The structural probe has a novel design, unattested in the parsing literature, the precise benefit of which is not immediately obvious. To explore whether syntactic probes would do better to make use of existing techniques, we compare the structural probe to a more traditional parser with an identical lightweight parameterisation. The parser outperforms structural probe on UUAS in seven of nine analysed languages, often by a substantial amount (e.g. by 11.1 points in English). Under a second less common metric, however, there is the opposite trend - the structural probe outperforms the parser. This begs the question: which metric should we prefer?