# Investigating grammatical abstraction in language models using few-shot learning of novel noun gender

arXiv: https://arxiv.org/abs/2403.10338 · HTML: https://mlsanity.com/paper/2403.10338/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment](https://mlsanity.com/paper/1909.08975/) (2019)
- [Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization](https://mlsanity.com/paper/2011.02417/) (2020)
- [Analyzing Gender Representation in Multilingual Models](https://mlsanity.com/paper/2204.09168/) (2022)
- [Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2010.05725/) (2020)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement](https://mlsanity.com/paper/2212.04523/) (2022)
- [Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rules](https://mlsanity.com/paper/2211.00153/) (2022)
- [Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study](https://mlsanity.com/paper/1909.04625/) (2019)

## Built on by (most cited first, 2 of 2)

- [Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts](https://mlsanity.com/paper/2606.30152/) (2026)
- [Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words](https://mlsanity.com/paper/2609.19887/) (2026)

## Abstract (arXiv)

Humans can learn a new word and infer its grammatical properties from very few examples. They have an abstract notion of linguistic properties like grammatical gender and agreement rules that can be applied to novel syntactic contexts and words. Drawing inspiration from psycholinguistics, we conduct a noun learning experiment to assess whether an LSTM and a decoder-only transformer can achieve human-like abstraction of grammatical gender in French. Language models were tasked with learning the gender of a novel noun embedding from a few examples in one grammatical agreement context and predicting agreement in another, unseen context. We find that both language models effectively generalise novel noun gender from one to two learning examples and apply the learnt gender across agreement contexts, albeit with a bias for the masculine gender category. Importantly, the few-shot updates were only applied to the embedding layers, demonstrating that models encode sufficient gender information within the word embedding space. While the generalisation behaviour of models suggests that they represent grammatical gender as an abstract category, like humans, further work is needed to explore the details of how exactly this is implemented. For a comparative perspective with human behaviour, we conducted an analogous one-shot novel noun gender learning experiment, which revealed that native French speakers, like language models, also exhibited a masculine gender bias and are not excellent one-shot learners either.