mlsanityData to 13 Oct 2026

What do RNN Language Models Learn about Filler-Gap Dependencies?

arXiv 1809.00042 ↗First posted 31 Aug 2018cs.CLcited by 86

Not on the map. Why?

Abstract (arXiv)

RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance filler-gap dependencies and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler-gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler-gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.

Key facts. Cited by 86 papers in the map (data through 13 Oct 2026). First posted on arXiv 31 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 14; 2022: 6; 2023: 15; 2024: 12; 2025: 21; 2026: 18.

86 papers.

Sort:

By area:

  1. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 129
  2. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  3. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  4. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  5. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 45
  6. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  7. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  8. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  9. 2024Mission: Impossible Language Modelscited by 31
  10. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 29
  11. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  12. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
38 more
  1. 2023Dissociating language and thought in large language modelscited by 27
  2. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  3. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  4. 2024Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairscited by 24
  5. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  6. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 23
  7. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 22
  8. 2023How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechcited by 22
  9. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  10. 2021How much pretraining data do language models need to learn syntax?cited by 21
  11. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  12. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilitiescited by 18
  13. 2021The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generationcited by 15
  14. 2023Evaluating Neural Language Models as Cognitive Models of Language Acquisitioncited by 15
  15. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  16. 2023Operationalising Representation in Natural Language Processingcited by 14
  17. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  18. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  19. 2023How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structurecited by 13
  20. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  21. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  22. 2021Effective Batching for Recurrent Neural Network Grammarscited by 12
  23. 2024Generalizations across filler-gap dependencies in neural language modelscited by 12
  24. 2023JCoLA: Japanese Corpus of Linguistic Acceptabilitycited by 11
  25. 2023Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semanticscited by 11
  26. 2025Large Language Models as Proxies for Theories of Human Linguistic Cognitioncited by 10
  27. 2025Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Modelscited by 9
  28. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  29. 2021Evaluating Document Coherence Modellingcited by 8
  30. 2021Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Modelscited by 7
  31. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 7
  32. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  33. 2025LingGym: How Far Are LLMs from Thinking Like Field Linguists?cited by 6
  34. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  35. 2025Construction Identification and Disambiguation Using BERT: A Case Study of NPNcited by 4
  36. 2022Representing Affect Information in Word Embeddingscited by 3
  37. 2023Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivitycited by 3
  38. 2026Clause-Internal or Clause-External? Testing Turkish Reflexive Binding in Adapted versus Chain of Thought Large Language Modelscited by 3

50 most cited of 86 All 86 (JSON)