# Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models

arXiv: https://arxiv.org/abs/2010.05725 · HTML: https://mlsanity.com/paper/2010.05725/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings](https://mlsanity.com/paper/2211.04928/) (2022)
- [Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation](https://mlsanity.com/paper/2201.01845/) (2022)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [Connecting degree and polarity: An artificial language learning study](https://mlsanity.com/paper/2109.06333/) (2021)
- [Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity](https://mlsanity.com/paper/2604.20789/) (2026)

## Abstract (arXiv)

Humans can learn structural properties about a word from minimal experience, and deploy their learned syntactic representations uniformly in different grammatical contexts. We assess the ability of modern neural language models to reproduce this behavior in English and evaluate the effect of structural supervision on learning outcomes. First, we assess few-shot learning capabilities by developing controlled experiments that probe models' syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. Second, we assess invariance properties of learned representation: the ability of a model to transfer syntactic generalizations from a base context (e.g., a simple declarative active-voice sentence) to a transformed context (e.g., an interrogative sentence). We test four models trained on the same dataset: an n-gram baseline, an LSTM, and two LSTM-variants trained with explicit structural supervision (Dyer et al.,2016; Charniak et al., 2016). We find that in most cases, the neural models are able to induce the proper syntactic generalizations after minimal exposure, often from just two examples during training, and that the two structurally supervised models generalize more accurately than the LSTM model. All neural models are able to leverage information learned in base contexts to drive expectations in transformed contexts, indicating that they have learned some invariance properties of syntax.