# Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction

arXiv: https://arxiv.org/abs/2105.06965 · HTML: https://mlsanity.com/paper/2105.06965/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution](https://mlsanity.com/paper/1909.00871/) (2019)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://mlsanity.com/paper/2010.10907/) (2020)
- [Contrastive Explanation: A Structural-Model Approach](https://mlsanity.com/paper/1811.03163/) (2018)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Causal Effects of Linguistic Properties](https://mlsanity.com/paper/2010.12919/) (2020)
- [Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models](https://mlsanity.com/paper/1909.10579/) (2019)

## Built on by (most cited first, 30 of 59)

- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions](https://mlsanity.com/paper/2207.14251/) (2022)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [Kernelized Concept Erasure](https://mlsanity.com/paper/2201.12191/) (2022)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models](https://mlsanity.com/paper/2203.09397/) (2022)
- [A Geometric Notion of Causal Probing](https://mlsanity.com/paper/2307.15054/) (2023)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://mlsanity.com/paper/2310.16142/) (2023)
- [Naturalistic Causal Probing for Morpho-Syntax](https://mlsanity.com/paper/2205.07043/) (2022)
- [Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs](https://mlsanity.com/paper/2404.09971/) (2024)
- [A Philosophical Introduction to Language Models - Part II: The Way Forward](https://mlsanity.com/paper/2405.03207/) (2024)
- [Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models](https://mlsanity.com/paper/2210.14328/) (2022)
- [Operationalising Representation in Natural Language Processing](https://mlsanity.com/paper/2306.08193/) (2023)
- [Gumbel Counterfactual Generation From Language Models](https://mlsanity.com/paper/2411.07180/) (2024)
- [How Reliable are Causal Probing Interventions?](https://mlsanity.com/paper/2408.15510/) (2024)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Analyzing Gender Representation in Multilingual Models](https://mlsanity.com/paper/2204.09168/) (2022)
- [Large Language Models as Proxies for Theories of Human Linguistic Cognition](https://mlsanity.com/paper/2502.07687/) (2025)

## Abstract (arXiv)

When language models process syntactically complex sentences, do they use their representations of syntax in a manner that is consistent with the grammar of the language? We propose AlterRep, an intervention-based method to address this question. For any linguistic feature of a given sentence, AlterRep generates counterfactual representations by altering how the feature is encoded, while leaving intact all other aspects of the original representation. By measuring the change in a model's word prediction behavior when these counterfactual representations are substituted for the original ones, we can draw conclusions about the causal effect of the linguistic feature in question on the model's behavior. We apply this method to study how BERT models of different sizes process relative clauses (RCs). We find that BERT variants use RC boundary information during word prediction in a manner that is consistent with the rules of English grammar; this RC boundary information generalizes to a considerable extent across different RC types, suggesting that BERT represents RCs as an abstract linguistic category.