# Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

arXiv: https://arxiv.org/abs/1903.00943 · HTML: https://mlsanity.com/paper/1903.00943/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2019.

## Built on by (most cited first, 24 of 24)

- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Language Models Use Monotonicity to Assess NPI Licensing](https://mlsanity.com/paper/2105.13818/) (2021)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [SLING: Sino Linguistic Evaluation of Large Language Models](https://mlsanity.com/paper/2210.11689/) (2022)
- [MELA: Multilingual Evaluation of Linguistic Acceptability](https://mlsanity.com/paper/2311.09033/) (2023)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024)
- [Language Modelling as a Multi-Task Problem](https://mlsanity.com/paper/2101.11287/) (2021)
- [Do Large Language Models Mirror Cognitive Language Processing?](https://mlsanity.com/paper/2402.18023/) (2024)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [Language Modeling with Latent Situations](https://mlsanity.com/paper/2212.10012/) (2022)
- [Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars](https://mlsanity.com/paper/2109.04939/) (2021)
- [Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models](https://mlsanity.com/paper/2109.11058/) (2021)
- [Using Integrated Gradients and Constituency Parse Trees to explain Linguistic Acceptability learnt by BERT](https://mlsanity.com/paper/2106.07349/) (2021)
- [Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation](https://mlsanity.com/paper/2201.01845/) (2022)
- [Composition, Attention, or Both?](https://mlsanity.com/paper/2210.12958/) (2022)
- [Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models](https://mlsanity.com/paper/2409.12435/) (2024)
- [FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition](https://mlsanity.com/paper/2403.00126/) (2024)
- [Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains](https://mlsanity.com/paper/2603.14400/) (2026)

## Abstract (arXiv)

State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a question that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus two models that represent hierarchical structure and deploy it in left-to-right processing: Recurrent Neural Network Grammars (RNNGs) (Dyer et al., 2016) and a incrementalized version of the Parsing-as-Language-Modeling configuration from Chariak et al., (2016). Models are tested on a diverse range of configurations for two classes of non-local grammatical dependencies in English---Negative Polarity licensing and Filler--Gap Dependencies. Using the same training data across models, we find that structurally-supervised models outperform the LSTM, with the RNNG demonstrating best results on both types of grammatical dependencies and even learning many of the Island Constraints on the filler--gap dependency. Structural supervision thus provides data efficiency advantages over purely string-based training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies.