# Improving Causal Interventions in Amnesic Probing with Mean Projection or LEACE

arXiv: https://arxiv.org/abs/2506.11673 · HTML: https://mlsanity.com/paper/2506.11673/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Kernelized Concept Erasure](https://mlsanity.com/paper/2201.12191/) (2022)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)

## Built on by (most cited first, 4 of 4)

- [The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASRrightarrowLLM Pipelines?](https://mlsanity.com/paper/2602.17598/) (2026)
- [Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension](https://mlsanity.com/paper/2608.10566/) (2026)
- [ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry](https://mlsanity.com/paper/2610.03125/) (2026)
- [Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulation](https://mlsanity.com/paper/2610.04676/) (2026)

## Abstract (arXiv)

Amnesic probing is a technique used to examine the influence of specific linguistic information on the behaviour of a model. This involves identifying and removing the relevant information and then assessing whether the model's performance on the main task changes. If the removed information is relevant, the model's performance should decline. The difficulty with this approach lies in removing only the target information while leaving other information unchanged. It has been shown that Iterative Nullspace Projection (INLP), a widely used removal technique, introduces random modifications to representations when eliminating target information. We demonstrate that Mean Projection (MP) and LEACE, two proposed alternatives, remove information in a more targeted manner, thereby enhancing the potential for obtaining behavioural explanations through Amnesic Probing.