Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
Not on the map. Why?
Abstract (arXiv)
Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at ≈33% sparsity.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
26 of 40 references.
References · 26
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
- 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
- 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
- 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 50
- 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
- 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
- 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
- 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 30
14 more
- 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inferencecited by 29
- 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
- 2026Training Language Models via Neural Cellular Automatacited by 15
- 2023Understanding the Inner Workings of Language Models Through Representation Dissimilaritycited by 12
- 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
- 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
- 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
- 2025Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervisioncited by 6
- 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
- 2025FOL-Traces: Verified First-Order Logic Reasoning Traces at Scalecited by 3
- 2026Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Modelscited by 3
- 2026Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Datacited by 3
- 2026Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Trainingcited by 2
- 2026Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.