# How Linguistics Learned to Stop Worrying and Love the Language Models

arXiv: https://arxiv.org/abs/2501.17047 · HTML: https://mlsanity.com/paper/2501.17047/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 356 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Builds on (most cited first, 30 of 78)

- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Theoretical Limitations of Self-Attention in Neural Sequence Models](https://mlsanity.com/paper/1906.06755/) (2019)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [On the Practical Computational Power of Finite Precision RNNs for Language Recognition](https://mlsanity.com/paper/1805.04908/) (2018)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Physics of Language Models: Part 1, Learning Hierarchical Language Structures](https://mlsanity.com/paper/2305.13673/) (2023)
- [Training Trajectories of Language Models Across Scales](https://mlsanity.com/paper/2212.09803/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Why are Sensitive Functions Hard for Transformers?](https://mlsanity.com/paper/2402.09963/) (2024)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [RNNs can generate bounded hierarchical languages with optimal memory](https://mlsanity.com/paper/2010.07515/) (2020)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [Language models align with human judgments on key grammatical constructions](https://mlsanity.com/paper/2402.01676/) (2024)

## Built on by (most cited first, 30 of 51)

- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions](https://mlsanity.com/paper/2505.16002/) (2025)
- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models](https://mlsanity.com/paper/2503.20850/) (2025)
- [LLMs as a synthesis between symbolic and distributed approaches to language](https://mlsanity.com/paper/2502.11856/) (2025)
- [Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition](https://mlsanity.com/paper/2502.04795/) (2025)
- [Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication](https://mlsanity.com/paper/2503.04395/) (2025)
- [Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning](https://mlsanity.com/paper/2508.01916/) (2025)
- [On the Acquisition of Shared Grammatical Representations in Bilingual Language Models](https://mlsanity.com/paper/2503.03962/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Constructions are Revealed in Word Distributions](https://mlsanity.com/paper/2503.06048/) (2025)
- [Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models](https://mlsanity.com/paper/2508.05803/) (2025)
- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible](https://mlsanity.com/paper/2510.07178/) (2025)
- [From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis](https://mlsanity.com/paper/2512.13491/) (2025)
- [Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations](https://mlsanity.com/paper/2601.03779/) (2026)
- [Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibility](https://mlsanity.com/paper/2503.17579/) (2025)
- [Language Models Identify Ambiguities and Exploit Loopholes](https://mlsanity.com/paper/2508.19546/) (2025)
- [WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives](https://mlsanity.com/paper/2510.09556/) (2025)
- [Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks](https://mlsanity.com/paper/2607.17624/) (2026)
- [The exponential distribution of the order of demonstrative, numeral, adjective and noun](https://mlsanity.com/paper/2502.06342/) (2025)
- [HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning](https://mlsanity.com/paper/2511.15574/) (2025)
- [Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking](https://mlsanity.com/paper/2602.17653/) (2026)
- [Perturbation: A simple and efficient adversarial tracer for representation learning in language models](https://mlsanity.com/paper/2603.23821/) (2026)
- [Syntactically-guided Information Maintenance in Sentence Comprehension](https://mlsanity.com/paper/2604.27468/) (2026)
- [Implicit Representations of Grammaticality in Language Models](https://mlsanity.com/paper/2605.05197/) (2026)
- [Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns](https://mlsanity.com/paper/2606.27460/) (2026)
- [Linear representations of grammaticality in neural language models](https://mlsanity.com/paper/2607.15175/) (2026)
- [Do Large Language Models know who did what to whom?](https://mlsanity.com/paper/2504.16884/) (2025)
- [Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences](https://mlsanity.com/paper/2601.09852/) (2026)

## Abstract (arXiv)

Language models can produce fluent, grammatical text. Nonetheless, some maintain that language models don't really learn language and also that, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the need for studying linguistic theory and structure. We argue that both extremes are wrong. LMs can contribute to fundamental questions about linguistic structure, language processing, and learning. They force us to rethink arguments and ways of thinking that have been foundational in linguistics. While they do not replace linguistic structure and theory, they serve as model systems and working proofs of concept for gradient, usage-based approaches to language. We offer an optimistic take on the relationship between language models and linguistics.