mlsanityData to 13 Oct 2026

Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages

arXiv 1903.06400 ↗First posted 15 Mar 2019cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language? Cross-linguistic comparisons of RNNs' syntactic performance (e.g., on subject-verb agreement prediction) are complicated by the fact that any two languages differ in multiple typological properties, as well as by differences in training corpus. We propose a paradigm that addresses these issues: we create synthetic versions of English, which differ from English in one or more typological parameters, and generate corpora for those languages based on a parsed English corpus. We report a series of experiments in which RNNs were trained to predict agreement features for verbs in each of those synthetic languages. Among other findings, (1) performance was higher in subject-verb-object order (as in English) than in subject-object-verb order (as in Japanese), suggesting that RNNs have a recency bias; (2) predicting agreement with both subject and object (polypersonal agreement) improves over predicting each separately, suggesting that underlying syntactic knowledge transfers across the two tasks; and (3) overt morphological case makes agreement prediction significantly easier, regardless of word order.

Key facts. Cited by 37 papers in the map (data through 13 Oct 2026). First posted on arXiv 15 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 10; 2022: 8; 2023: 10; 2024: 3; 2025: 3; 2026: 3.

37 papers.

Sort:

By area:

  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 244
  3. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  4. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  5. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  6. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  7. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  8. 2021Examining the Inductive Bias of Neural Language Models with Artificial Languagescited by 40
  9. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 35
  10. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  11. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  12. 2023Dissociating language and thought in large language modelscited by 27
25 more
  1. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  2. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
  3. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  4. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 20
  5. 2023A Cross-Linguistic Pressure for Uniform Information Density in Word Ordercited by 16
  6. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  7. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  8. 2023Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distributioncited by 10
  9. 2022Lexical Generalization Improves with Larger Models and Longer Trainingcited by 9
  10. 2021On the Difficulty of Translating Free-Order Case-Marking Languagescited by 8
  11. 2023Towards Bridging the Digital Language Dividecited by 7
  12. 2021Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuningcited by 5
  13. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  14. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Studycited by 5
  15. 2026Pretraining Recurrent Networks without Recurrencecited by 5
  16. 2025Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languagescited by 4
  17. 2021Towards Zero-shot Language Modelingcited by 3
  18. 2021Leveraging Transformers for StarCraft Macromanagement Predictioncited by 3
  19. 2024Cookbook: A framework for improving LLM generative abilities via programmatic data generating templatescited by 3
  20. 2021Enhancing the Transformer Decoder with Transition-based Syntaxcited by 1
  21. 2024Training an NLP Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposalcited by 1
  22. 2026Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspectivecited by 1
  23. 2022Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?cited by 0
  24. 2023Does Character-level Information Always Improve DRS-based Semantic Parsing?cited by 0
  25. 2026Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Modelscited by 0