# Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

arXiv: https://arxiv.org/abs/2006.00995 · HTML: https://mlsanity.com/paper/2006.00995/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 14 of 14)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [The Low-Dimensional Linear Geometry of Contextualized Word Representations](https://mlsanity.com/paper/2105.07109/) (2021)
- [Unit Testing for Concepts in Neural Networks](https://mlsanity.com/paper/2208.10244/) (2022)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)
- [Operationalising Representation in Natural Language Processing](https://mlsanity.com/paper/2306.08193/) (2023)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations](https://mlsanity.com/paper/2101.11492/) (2021)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [Probing with Noise: Unpicking the Warp and Weft of Embeddings](https://mlsanity.com/paper/2210.12206/) (2022)
- [Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?](https://mlsanity.com/paper/2106.10622/) (2021)
- [Visualizing the Relationship Between Encoded Linguistic Information and Task Performance](https://mlsanity.com/paper/2203.15860/) (2022)
- [Does the Model Use the Feature? Separating Steering from Mechanism in LLMs](https://mlsanity.com/paper/2610.07270/) (2026)

## Abstract (arXiv)

A growing body of work makes use of probing to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, e.g. is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.