# Universal linguistic inductive biases via meta-learning

arXiv: https://arxiv.org/abs/2006.16324 · HTML: https://mlsanity.com/paper/2006.16324/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 29 of 29)

- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning](https://mlsanity.com/paper/2101.06223/) (2021)
- [Modeling rapid language learning by distilling Bayesian priors into artificial neural networks](https://mlsanity.com/paper/2305.14701/) (2023)
- [Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases](https://mlsanity.com/paper/2502.19249/) (2025)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)
- [A Property Induction Framework for Neural Language Models](https://mlsanity.com/paper/2205.06910/) (2022)
- [Meta-Learning Neural Mechanisms rather than Bayesian Priors](https://mlsanity.com/paper/2503.16048/) (2025)
- [SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation](https://mlsanity.com/paper/2310.00796/) (2023)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning](https://mlsanity.com/paper/2204.01437/) (2022)
- [Instilling Inductive Biases with Subnetworks](https://mlsanity.com/paper/2310.10899/) (2023)
- [Distilling Symbolic Priors for Concept Learning into Neural Networks](https://mlsanity.com/paper/2402.07035/) (2024)
- [Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks](https://mlsanity.com/paper/2502.20237/) (2025)
- [Distinct Computations Emerge From Compositional Curricula in In-Context Learning](https://mlsanity.com/paper/2506.13253/) (2025)
- [Transfer of Structural Knowledge from Synthetic Languages](https://mlsanity.com/paper/2505.15769/) (2025)
- [Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2508.14285/) (2025)
- [Meta-Learning Theory-Informed Inductive Biases using Deep Kernel Gaussian Processes](https://mlsanity.com/paper/2509.24919/) (2025)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Meta-learning as a principle for human-like visual representations](https://mlsanity.com/paper/2606.28399/) (2026)

## Abstract (arXiv)

How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it is unclear which inductive biases can explain observed patterns in language acquisition. To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases. This framework disentangles universal inductive biases, which are encoded in the initial values of a neural network's parameters, from non-universal factors, which the neural network must learn from data in a given language. The initial state that encodes the inductive biases is found with meta-learning, a technique through which a model discovers how to acquire new languages more easily via exposure to many possible languages. By controlling the properties of the languages that are used during meta-learning, we can control the inductive biases that meta-learning imparts. We demonstrate this framework with a case study based on syllable structure. First, we specify the inductive biases that we intend to give our model, and then we translate those inductive biases into a space of languages from which a model can meta-learn. Finally, using existing analysis techniques, we verify that our approach has imparted the linguistic inductive biases that it was intended to impart.