# Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rules

arXiv: https://arxiv.org/abs/2211.00153 · HTML: https://mlsanity.com/paper/2211.00153/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 8 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 6 of 6)

- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)

## Built on by (most cited first, 6 of 6)

- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing](https://mlsanity.com/paper/2310.11923/) (2023)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [PatternGPT :A Pattern-Driven Framework for Large Language Model Text Generation](https://mlsanity.com/paper/2307.00470/) (2023)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)

## Abstract (arXiv)

LSTMs trained on next-word prediction can accurately perform linguistic tasks that require tracking long-distance syntactic dependencies. Notably, model accuracy approaches human performance on number agreement tasks (Gulordava et al., 2018). However, we do not have a mechanistic understanding of how LSTMs perform such linguistic tasks. Do LSTMs learn abstract grammatical rules, or do they rely on simple heuristics? Here, we test gender agreement in French which requires tracking both hierarchical syntactic structures and the inherent gender of lexical units. Our model is able to reliably predict long-distance gender agreement in two subject-predicate contexts: noun-adjective and noun-passive-verb agreement. The model showed more inaccuracies on plural noun phrases with gender attractors compared to singular cases, suggesting a reliance on clues from gendered articles for agreement. Overall, our study highlights key ways in which LSTMs deviate from human behaviour and questions whether LSTMs genuinely learn abstract syntactic rules and categories. We propose using gender agreement as a useful probe to investigate the underlying mechanisms, internal representations, and linguistic capabilities of LSTM language models.