# An information theoretic view on selecting linguistic probes

arXiv: https://arxiv.org/abs/2009.07364 · HTML: https://mlsanity.com/paper/2009.07364/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Exploring the Role of BERT Token Representations to Explain Sentence Probing Results](https://mlsanity.com/paper/2104.01477/) (2021)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [Operationalising Representation in Natural Language Processing](https://mlsanity.com/paper/2306.08193/) (2023)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [Strong hallucinations from negation and how to fix them](https://mlsanity.com/paper/2402.10543/) (2024)
- [Interventional Probing in High Dimensions: An NLI Case Study](https://mlsanity.com/paper/2304.10346/) (2023)
- [On the data requirements of probing](https://mlsanity.com/paper/2202.12801/) (2022)
- [Quantifying the Task-Specific Information in Text-Based Classifications](https://mlsanity.com/paper/2110.08931/) (2021)
- [On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL](https://mlsanity.com/paper/2404.02389/) (2024)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings](https://mlsanity.com/paper/2301.08998/) (2023)
- [Semantic Composition in Visually Grounded Language Models](https://mlsanity.com/paper/2305.16328/) (2023)
- [Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data](https://mlsanity.com/paper/2407.13765/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations](https://mlsanity.com/paper/2506.21682/) (2025)

## Abstract (arXiv)

There is increasing interest in assessing the linguistic knowledge encoded in neural representations. A popular approach is to attach a diagnostic classifier -- or "probe" -- to perform supervised classification from internal representations. However, how to select a good probe is in debate. Hewitt and Liang (2019) showed that a high performance on diagnostic classification itself is insufficient, because it can be attributed to either "the representation being rich in knowledge", or "the probe learning the task", which Pimentel et al. (2020) challenged. We show this dichotomy is valid information-theoretically. In addition, we find that the methods to construct and select good probes proposed by the two papers, *control task* (Hewitt and Liang, 2019) and *control function* (Pimentel et al., 2020), are equivalent -- the errors of their approaches are identical (modulo irrelevant terms). Empirically, these two selection criteria lead to results that highly agree with each other.