# Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training

arXiv: https://arxiv.org/abs/2506.22638 · HTML: https://mlsanity.com/paper/2506.22638/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 33 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 8 of 8)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Streamlining Redundant Layers to Compress Large Language Models](https://mlsanity.com/paper/2403.19135/) (2024)

## Built on by (most cited first, 9 of 9)

- [Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning](https://mlsanity.com/paper/2510.02091/) (2025)
- [Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures](https://mlsanity.com/paper/2601.19928/) (2026)
- [What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis](https://mlsanity.com/paper/2602.12395/) (2026)
- [On the Limits of Layer Pruning for Generative Reasoning in Large Language Models](https://mlsanity.com/paper/2602.01997/) (2026)
- [Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training](https://mlsanity.com/paper/2607.01232/) (2026)
- [Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4](https://mlsanity.com/paper/2603.08747/) (2026)
- [Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models](https://mlsanity.com/paper/2604.08284/) (2026)
- [What Does Layer-Importance Reveal About Transformers and State-Space Models?](https://mlsanity.com/paper/2609.16537/) (2026)
- [Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining](https://mlsanity.com/paper/2610.01165/) (2026)

## Abstract (arXiv)

Large language models improve at math after instruction tuning, reinforcement learning, or knowledge distillation. We ask whether these gains come from major changes in the transformer layers or from smaller adjustments that keep the original structure. Using layer-wise ablation on base and trained variants, we find that math reasoning depends on a few critical layers, which stay important across all post-training methods. Removing these layers reduces math accuracy by as much as 80%, whereas factual recall tasks only show relatively smaller drops. This suggests that specialized layers for mathematical tasks form during pre-training and remain stable afterward. As measured by Normalized Mutual Information (NMI), we find that near these critical layers, tokens drift from their original syntactic clusters toward representations aligned with tokens less syntactically related but potentially more useful for downstream task.