mlsanityData to 14 Oct 2026

A Survey on LLM Mid-Training

arXiv 2510.23081 ↗First posted 27 Oct 2025cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage that bridges pre-training and post-training. Mid-training is distinguished by its use of intermediate data and computational resources, systematically enhancing specified capabilities such as mathematics, coding, reasoning, and long-context extension, while maintaining foundational competencies. This survey provides a formal definition of mid-training for large language models (LLMs) and investigates optimization frameworks that encompass data curation, training strategies, and model architecture optimization. We analyze mainstream model implementations in the context of objective-driven interventions, illustrating how mid-training serves as a distinct and critical stage in the progressive development of LLM capabilities. By clarifying the unique contributions of mid-training, this survey offers a comprehensive taxonomy and actionable insights, supporting future research and innovation in the advancement of LLMs.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 77 of its 90 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

77 of 90 references.

References · 77

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  13. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  14. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  15. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  16. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  17. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  18. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  19. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  20. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  21. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  22. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  23. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  24. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  25. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  26. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  27. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  28. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  29. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  30. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  31. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  32. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  33. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  34. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  35. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  36. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  37. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  38. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 20.

21 papers.

Sort:

By area:

  1. 2026daVinci-Dev: Agent-native Mid-training for Software Engineeringcited by 18
  2. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  3. 2026Thoth: Mid-Training Bridges LLMs to Time Series Understandingcited by 4
  4. 2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
  5. 2026MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selectioncited by 3
  6. 2026Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Modelscited by 2
  7. 2026State2State: Environment-Derived Mid-Training for LLM Agentscited by 2
  8. 2026Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Modelscited by 1
  9. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  10. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Modelscited by 1
  11. 2026GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselectioncited by 1
  12. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
9 more
  1. 2026MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resourcescited by 1
  2. 2026MidTool: Mid-training Data Synthesis for Agentic Tool Usecited by 1
  3. 2026Stability as a Liability:Systematic Breakdown of Linguistic Structure in LLMscited by 0
  4. 2026Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithmcited by 0
  5. 2026Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cyclecited by 0
  6. 2026Wasted large language models: A life cycle thinking approachcited by 0
  7. 2026Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Trainingcited by 0
  8. 2026MiST: Mid-Training LLMs for Cybersecuritycited by 0
  9. 2026TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Trainingcited by 0