# A Survey on LLM Mid-Training

arXiv: https://arxiv.org/abs/2510.23081 · HTML: https://mlsanity.com/paper/2510.23081/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 77 of its 90 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 30 of 77)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)

## Built on by (most cited first, 21 of 21)

- [daVinci-Dev: Agent-native Mid-training for Software Engineering](https://mlsanity.com/paper/2601.18418/) (2026)
- [Efficient Context Scaling with LongCat ZigZag Attention](https://mlsanity.com/paper/2512.23966/) (2025)
- [Thoth: Mid-Training Bridges LLMs to Time Series Understanding](https://mlsanity.com/paper/2603.01042/) (2026)
- [Can RL Improve Generalization of LLM Agents? An Empirical Study](https://mlsanity.com/paper/2603.12011/) (2026)
- [MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection](https://mlsanity.com/paper/2605.30288/) (2026)
- [Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models](https://mlsanity.com/paper/2607.12463/) (2026)
- [State2State: Environment-Derived Mid-Training for LLM Agents](https://mlsanity.com/paper/2608.04934/) (2026)
- [Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models](https://mlsanity.com/paper/2601.06911/) (2026)
- [Buy versus Build an LLM: A Decision Framework for Governments](https://mlsanity.com/paper/2602.13033/) (2026)
- [Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models](https://mlsanity.com/paper/2605.08472/) (2026)
- [GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection](https://mlsanity.com/paper/2605.28534/) (2026)
- [A Primer in Post-Training Reasoning Data: What We Know About How It Works](https://mlsanity.com/paper/2606.02113/) (2026)
- [MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources](https://mlsanity.com/paper/2606.25832/) (2026)
- [MidTool: Mid-training Data Synthesis for Agentic Tool Use](https://mlsanity.com/paper/2608.20314/) (2026)
- [Stability as a Liability:Systematic Breakdown of Linguistic Structure in LLMs](https://mlsanity.com/paper/2601.18588/) (2026)
- [Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm](https://mlsanity.com/paper/2605.10640/) (2026)
- [Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cycle](https://mlsanity.com/paper/2607.22877/) (2026)
- [Wasted large language models: A life cycle thinking approach](https://mlsanity.com/paper/2608.17055/) (2026)
- [Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training](https://mlsanity.com/paper/2609.09081/) (2026)
- [MiST: Mid-Training LLMs for Cybersecurity](https://mlsanity.com/paper/2609.18496/) (2026)
- [TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training](https://mlsanity.com/paper/2610.04973/) (2026)

## Abstract (arXiv)

Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage that bridges pre-training and post-training. Mid-training is distinguished by its use of intermediate data and computational resources, systematically enhancing specified capabilities such as mathematics, coding, reasoning, and long-context extension, while maintaining foundational competencies. This survey provides a formal definition of mid-training for large language models (LLMs) and investigates optimization frameworks that encompass data curation, training strategies, and model architecture optimization. We analyze mainstream model implementations in the context of objective-driven interventions, illustrating how mid-training serves as a distinct and critical stage in the progressive development of LLM capabilities. By clarifying the unique contributions of mid-training, this survey offers a comprehensive taxonomy and actionable insights, supporting future research and innovation in the advancement of LLMs.