mlsanityData to 7 Oct 2026

Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases

arXiv 2502.19249 ↗First posted 26 Feb 2025cs.CL cs.AI cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Pretraining language models on formal language can improve their acquisition of natural language. Which features of the formal language impart an inductive bias that leads to effective transfer? Drawing on insights from linguistics and complexity theory, we hypothesize that effective transfer occurs when two conditions are met: the formal language should capture the dependency structures present in natural language, and it should remain within the computational limitations of the model architecture. We experiment with pre-pretraining (training on formal language before natural languages) on transformers and find that formal languages capturing hierarchical dependencies indeed enable language models to achieve lower loss on natural language and better linguistic generalization compared to other formal languages. We also find modest support for the hypothesis that the formal language should fall within the computational limitations of the architecture. Strikingly, pre-pretraining reduces loss more efficiently than training on a matched amount of natural language. For a 1B-parameter language model trained on roughly 1.6B tokens of natural language, pre-pretraining achieves the same loss and better linguistic generalization with a 33% smaller token budget. Finally, we also give mechanistic evidence of transfer from formal to natural language: attention heads acquired during pre-pretraining remain crucial for the model's performance on syntactic evaluations.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

50 of 80 references.

References · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  10. 2019CamemBERT: a Tasty French Language Modelcited by 368
  11. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  12. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
38 more
  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  2. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  3. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  4. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  5. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  6. 2021Thinking Like Transformerscited by 182
  7. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  8. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  9. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  10. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  11. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  12. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  13. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  14. 2022A Logic for Expressing Log-Precision Transformerscited by 67
  15. 2020A Formal Hierarchy of RNN Architecturescited by 61
  16. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 47
  17. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  18. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  19. 2019LSTM Networks Can Perform Dynamic Countingcited by 45
  20. 2024A Formal Framework for Understanding Length Generalization in Transformerscited by 44
  21. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  22. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  23. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  24. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  25. 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
  26. 2020Universal linguistic inductive biases via meta-learningcited by 28
  27. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  28. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  29. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  30. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  31. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  32. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  33. 2024Training Neural Networks as Recognizers of Formal Languagescited by 23
  34. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  35. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  36. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8
  37. 2024Transformer verbatim in-context retrieval across time and scalecited by 3
  38. 2024Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 28.

36 papers.

Sort:

By area:

  1. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  2. 2026Training Language Models via Neural Cellular Automatacited by 15
  3. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  4. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  5. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  6. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  7. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 4
  8. 2025Unraveling Syntax: Language Modeling and the Substructure of Grammarscited by 4
  9. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  10. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  11. 2026Context-Free Recognition with Transformerscited by 3
  12. 2026Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Datacited by 3
24 more
  1. 2025Is In-Context Learning Learning?cited by 2
  2. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  3. 2026Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibilitycited by 2
  4. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  5. 2026A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Modelscited by 1
  6. 2026Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scalecited by 1
  7. 2026Efficient Pre-Training with Token Superpositioncited by 1
  8. 2026Language Acquisition Device in Large Language Modelscited by 1
  9. 2026Emergent Capabilities Arise Randomly from Learning Sparse Attention Patternscited by 1
  10. 2026Structural priors for data-efficient language learningcited by 1
  11. 2026Self-Play Pretraining with Zero Datacited by 1
  12. 2026Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequencescited by 0
  13. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  14. 2026Rubato: Transcribing Piano Music with Timestampscited by 0
  15. 2026An In-Vitro Study on Cross-Lingual Generalization in Language Modelscited by 0
  16. 2026Linear representations of grammaticality in neural language modelscited by 0
  17. 2026Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Trainingcited by 0
  18. 2026Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languagescited by 0
  19. 2026Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMscited by 0
  20. 2026Length Generalization for Transformers via Compressioncited by 0
  21. 2026Procedural Pretraining for Molecular Property Predictioncited by 0
  22. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0
  23. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0
  24. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0