# Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2411.12580 · HTML: https://mlsanity.com/paper/2411.12580/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization](https://mlsanity.com/paper/2405.15071/) (2024)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [To Code, or Not To Code? Exploring Impact of Code in Pre-training](https://mlsanity.com/paper/2408.10914/) (2024)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Evidence from counterfactual tasks supports emergent analogical reasoning in large language models](https://mlsanity.com/paper/2404.13070/) (2024)

## Built on by (most cited first, 30 of 41)

- [SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement](https://mlsanity.com/paper/2504.07934/) (2025)
- [Organize the Web: Constructing Domains Enhances Pre-Training Data Curation](https://mlsanity.com/paper/2502.10341/) (2025)
- [Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective](https://mlsanity.com/paper/2412.14135/) (2024)
- [Rethinking Reflection in Pre-Training](https://mlsanity.com/paper/2504.04022/) (2025)
- [On the generalization of language models from in-context learning and finetuning: a controlled study](https://mlsanity.com/paper/2505.00661/) (2025)
- [Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents](https://mlsanity.com/paper/2509.03581/) (2025)
- [Data Mixing Can Induce Phase Transitions in Knowledge Acquisition](https://mlsanity.com/paper/2505.18091/) (2025)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [Are Language Models Consequentialist or Deontological Moral Reasoners?](https://mlsanity.com/paper/2505.21479/) (2025)
- [Influence Functions for Efficient Data Selection in Reasoning](https://mlsanity.com/paper/2510.06108/) (2025)
- [Procedural Pretraining: Warming Up Language Models with Abstract Data](https://mlsanity.com/paper/2601.21725/) (2026)
- [Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning](https://mlsanity.com/paper/2505.22308/) (2025)
- [Enhancing Training Data Attribution with Representational Optimization](https://mlsanity.com/paper/2505.18513/) (2025)
- [Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions](https://mlsanity.com/paper/2505.19949/) (2025)
- [IF-GUIDE: Influence Function-Guided Detoxification of LLMs](https://mlsanity.com/paper/2506.01790/) (2025)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets](https://mlsanity.com/paper/2608.11233/) (2026)
- [AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection](https://mlsanity.com/paper/2505.07293/) (2025)
- [Infusion: Shaping Model Behavior by Editing Training Data via Influence Functions](https://mlsanity.com/paper/2602.09987/) (2026)
- [Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data](https://mlsanity.com/paper/2506.00469/) (2025)
- [A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)](https://mlsanity.com/paper/2602.14696/) (2026)
- [Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attribution](https://mlsanity.com/paper/2507.06547/) (2025)
- [Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics](https://mlsanity.com/paper/2508.11017/) (2025)
- [Test of Time: Rethinking Temporal Signal of Benchmark Contamination](https://mlsanity.com/paper/2509.00072/) (2025)
- [AIR: Post-training Data Selection for Reasoning via Attention Head Influence](https://mlsanity.com/paper/2512.13279/) (2025)
- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [Bergson: An Open Source Library for Data Attribution](https://mlsanity.com/paper/2606.11660/) (2026)
- [Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling](https://mlsanity.com/paper/2501.13779/) (2025)
- [Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning](https://mlsanity.com/paper/2502.20620/) (2025)

## Abstract (arXiv)

The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.