# Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment

arXiv: https://arxiv.org/abs/1909.08975 · HTML: https://mlsanity.com/paper/1909.08975/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition](https://mlsanity.com/paper/2407.00886/) (2024)
- [The Birth of Bias: A case study on the evolution of gender bias in an English language model](https://mlsanity.com/paper/2207.10245/) (2022)
- [Evaluating the Deductive Competence of Large Language Models](https://mlsanity.com/paper/2309.05452/) (2023)
- [Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans](https://mlsanity.com/paper/2110.07240/) (2021)
- [Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement](https://mlsanity.com/paper/2212.04523/) (2022)
- [Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation](https://mlsanity.com/paper/2505.08546/) (2025)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue](https://mlsanity.com/paper/2311.13061/) (2023)
- [Attention vs non-attention for a Shapley-based explanation method](https://mlsanity.com/paper/2104.12424/) (2021)
- [The Unheard Alternative: Contrastive Explanations for Speech-to-Text Models](https://mlsanity.com/paper/2509.26543/) (2025)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures](https://mlsanity.com/paper/2603.17952/) (2026)
- [Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models](https://mlsanity.com/paper/2311.16978/) (2023)

## Abstract (arXiv)

Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we propose a generalisation of Contextual Decomposition (GCD). In particular, this setup enables us to accurately distil which part of a prediction stems from semantic heuristics, which part truly emanates from syntactic cues and which part arise from the model biases themselves instead. We investigate this technique on tasks pertaining to syntactic agreement and co-reference resolution and discover that the model strongly relies on a default reasoning effect to perform these tasks.