# What if This Modified That? Syntactic Interventions via Counterfactual Embeddings

arXiv: https://arxiv.org/abs/2105.14002 · HTML: https://mlsanity.com/paper/2105.14002/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [A Tale of a Probe and a Parser](https://mlsanity.com/paper/2005.01641/) (2020)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)

## Built on by (most cited first, 30 of 39)

- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model](https://mlsanity.com/paper/2306.05720/) (2023)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [Unit Testing for Concepts in Neural Networks](https://mlsanity.com/paper/2208.10244/) (2022)
- [Naturalistic Causal Probing for Morpho-Syntax](https://mlsanity.com/paper/2205.07043/) (2022)
- [Evaluating Neural Language Models as Cognitive Models of Language Acquisition](https://mlsanity.com/paper/2310.20093/) (2023)
- [A Review of the Role of Causality in Developing Trustworthy AI Systems](https://mlsanity.com/paper/2302.06975/) (2023)
- [Operationalising Representation in Natural Language Processing](https://mlsanity.com/paper/2306.08193/) (2023)
- [When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes](https://mlsanity.com/paper/2204.09722/) (2022)
- [Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number](https://mlsanity.com/paper/2310.15151/) (2023)
- [Uncovering Intermediate Variables in Transformers using Circuit Probing](https://mlsanity.com/paper/2311.04354/) (2023)
- [How Reliable are Causal Probing Interventions?](https://mlsanity.com/paper/2408.15510/) (2024)
- [Why Linguistics Will Thrive in the 21st Century: A Reply to Piantadosi (2023)](https://mlsanity.com/paper/2308.03228/) (2023)
- [More than Correlation: Do Large Language Models Learn Causal Representations of Space?](https://mlsanity.com/paper/2312.16257/) (2023)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://mlsanity.com/paper/2506.16678/) (2025)
- [Analyzing the Inner Workings of Transformers in Compositional Generalization](https://mlsanity.com/paper/2502.15277/) (2025)
- [A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution](https://mlsanity.com/paper/2404.01921/) (2024)
- [Prototype Based Classification from Hierarchy to Fairness](https://mlsanity.com/paper/2205.13997/) (2022)
- [Competence-Based Analysis of Language Models](https://mlsanity.com/paper/2303.00333/) (2023)
- [From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?](https://mlsanity.com/paper/2509.17280/) (2025)
- [Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings](https://mlsanity.com/paper/2301.08998/) (2023)
- [On Modifying a Neural Network's Perception](https://mlsanity.com/paper/2303.02655/) (2023)

## Abstract (arXiv)

Neural language models exhibit impressive performance on a variety of tasks, but their internal reasoning may be difficult to understand. Prior art aims to uncover meaningful properties within model representations via probes, but it is unclear how faithfully such probes portray information that the models actually use. To overcome such limitations, we propose a technique, inspired by causal analysis, for generating counterfactual embeddings within models. In experiments testing our technique, we produce evidence that suggests some BERT-based models use a tree-distance-like representation of syntax in downstream prediction tasks.