# Examining the Inductive Bias of Neural Language Models with Artificial Languages arXiv: https://arxiv.org/abs/2106.01044 ยท HTML: https://mlsanity.com/paper/2106.01044/ ## Key facts - Cited by 40 papers in the map (data through 14 Oct 2026). - Builds on 8 of its 21 references that are in the map. - First posted on arXiv 2 Jun 2021. ## Builds on (most cited first, 8 of 8) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019) - [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018) - [What Kind of Language Is Hard to Language-Model?](https://mlsanity.com/paper/1906.04726/) (2019) - [Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages](https://mlsanity.com/paper/1903.06400/) (2019) - [Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks](https://mlsanity.com/paper/1802.09091/) (2018) - [Tree-structured composition in neural networks without tree-structured architectures](https://mlsanity.com/paper/1506.04834/) (2015) - [The Galactic Dependencies Treebanks: Getting More Data by Synthesizing New Languages](https://mlsanity.com/paper/1710.03838/) (2017) ## Built on by (most cited first, 30 of 40) - [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023) - [Injecting structural hints: Using language models to study inductive biases in language learning](https://mlsanity.com/paper/2304.13060/) (2023) - [Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models](https://mlsanity.com/paper/2203.10326/) (2022) - [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023) - [Can Language Models Learn Typologically Implausible Languages?](https://mlsanity.com/paper/2502.12317/) (2025) - [What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages](https://mlsanity.com/paper/2406.04289/) (2024) - [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022) - [A Cross-Linguistic Pressure for Uniform Information Density in Word Order](https://mlsanity.com/paper/2306.03734/) (2023) - [Compositional Evaluation on Japanese Textual Entailment and Similarity](https://mlsanity.com/paper/2208.04826/) (2022) - [Benchmarking Compositionality with Formal Languages](https://mlsanity.com/paper/2208.08195/) (2022) - [Emergent Word Order Universals from Cognitively-Motivated Language Models](https://mlsanity.com/paper/2402.12363/) (2024) - [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022) - [Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution](https://mlsanity.com/paper/2310.14840/) (2023) - [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025) - [Mechanistic evaluation of Transformers and state space models](https://mlsanity.com/paper/2505.15105/) (2025) - [The Role of Learning Algorithms in Collective Action](https://mlsanity.com/paper/2405.06582/) (2024) - [NeLLCom-X: A Comprehensive Neural-Agent Framework to Simulate Language Learning and Group Communication](https://mlsanity.com/paper/2407.13999/) (2024) - [Towards Bridging the Digital Language Divide](https://mlsanity.com/paper/2307.13405/) (2023) - [Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience](https://mlsanity.com/paper/2408.12664/) (2024) - [Word Order Matters when you Increase Masking](https://mlsanity.com/paper/2211.04427/) (2022) - [Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages](https://mlsanity.com/paper/2510.12722/) (2025) - [Reinforcement Learning Fine-tuning of Language Models is Biased Towards More Extractable Features](https://mlsanity.com/paper/2311.04046/) (2023) - [The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang](https://mlsanity.com/paper/2509.00425/) (2025) - [Using Artificial French Data to Understand the Emergence of Gender Bias in Transformer Language Models](https://mlsanity.com/paper/2310.15852/) (2023) - [The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translation](https://mlsanity.com/paper/2403.18031/) (2024) - [Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale](https://mlsanity.com/paper/2603.06592/) (2026) - [Can Language Models Induce Grammatical Knowledge from Indirect Evidence?](https://mlsanity.com/paper/2410.06022/) (2024) - [Is Structure Dependence Shaped for Efficient Communication?: A Case Study on Coordination](https://mlsanity.com/paper/2410.10556/) (2024) - [Rethinking Memorization Measures and their Implications in Large Language Models](https://mlsanity.com/paper/2507.14777/) (2025) - [Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization](https://mlsanity.com/paper/2601.12598/) (2026) ## Abstract (arXiv) Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These languages are constructed to allow us to create parallel corpora across languages that differ only in the typological feature being investigated, such as word order. We then use them to train and test language models. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models. Using this method, we find that commonly used neural architectures exhibit different inductive biases: LSTMs display little preference with respect to word ordering, while transformers display a clear preference for some orderings over others. Further, we find that neither the inductive bias of the LSTM nor that of the transformer appears to reflect any tendencies that we see in attested natural languages.