mlsanityData to 14 Oct 2026

Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence

arXiv 2405.15750 ↗First posted 24 May 2024cs.CL cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpora that target a wide range of linguistic phenomena. Our results show that while transformers are better qua LMs (as measured by perplexity), both models perform equally and surprisingly well on linguistic generalization measures, suggesting that they are capable of generalizing from indirect evidence.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

35 of 54 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Neural Network Acceptability Judgmentscited by 958
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  5. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  8. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  9. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  10. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  11. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  12. 2018Colorless green recurrent networks dream hierarchicallycited by 213
23 more
  1. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  2. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  3. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  4. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  5. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  7. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  8. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  9. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  10. 2021Lower Perplexity is Not Always Human-Likecited by 52
  11. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  12. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  13. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  14. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  15. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  16. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 31
  17. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  18. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  19. 2018Verb Argument Structure Alternations in Word and Sentence Embeddingscited by 25
  20. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  21. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  22. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  23. 2021Language Modelling as a Multi-Task Problemcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 19.

35 papers.

Sort:

By area:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  3. 2025Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Modelscited by 18
  4. 2024Generalizations across filler-gap dependencies in neural language modelscited by 12
  5. 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
  6. 2024A systematic framework for generating novel experimental hypotheses from language modelscited by 9
  7. 2025Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Modelscited by 9
  8. 2025On the Acquisition of Shared Grammatical Representations in Bilingual Language Modelscited by 7
  9. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  10. 2026Humans and transformer LMs: Abstraction drives language learningcited by 2
  11. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  12. 2024Can Language Models Induce Grammatical Knowledge from Indirect Evidence?cited by 1
23 more
  1. 2025The Structural Sources of Verb Meaning Revisited: Large Language Models Display Syntactic Bootstrappingcited by 1
  2. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  3. 2026Function Words as Statistical Cues for Language Learningcited by 1
  4. 2026A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Modelscited by 1
  5. 2026Differences in Typological Alignment in Language Models' Treatment of Differential Argument Markingcited by 1
  6. 2026What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependenciescited by 1
  7. 2026Cross-Modal Taxonomic Generalization in (Vision-) Language Modelscited by 1
  8. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  9. 2026Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretrainingcited by 1
  10. 2024Black Big Boxes: Tracing Adjective Order Preferences in Large Language Modelscited by 0
  11. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  12. 2025Large language models have learned to use languagecited by 0
  13. 2025Large language models are not about natural languagecited by 0
  14. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  15. 2026Are Language Models Models?cited by 0
  16. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  17. 2026You Can't Fight in Here! This is BBS!cited by 0
  18. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  19. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  20. 2026Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networkscited by 0
  21. 2026Exposure is Optional: Learning Unlike Coordination in Language Modelscited by 0
  22. 2026Language Models Generalize to Human-like Word Order Preferencescited by 0
  23. 2026Disentangling Statistical Preemption from Entrenchment in Language Models' Avoidance of Overgeneralizationcited by 0