Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
Not on the map. Why?
Abstract (arXiv)
Mid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it. In a controlled logical-reasoning setting (Qwen3-8B-Base, with a 4B replication; five semantically rule-disjoint KOR-Bench domains) we train 30 allocations spanning the five-domain simplex, 24 sweep configurations plus six withheld from the fit, at five seeds each. Three findings emerge. First, every domain has an interior coverage optimum: the moderate band (10%-40%) is best for all five domains, and a calibrated permutation test for quadratic interiority gives P≈0.010; the fitted mid-training-only curves, with 8B peaks between 9.9% and 35.1%, reproduce for curve shape but not peak location. Second, the gaps survive a fixed-budget alignment pass: compensatory SFT raises 116/120 cells (mean +4.32%) yet bridges 0/240 pairs at a 5% threshold and 30/240 at a 10% ratio, an equal-budget uniform control behaves almost identically, and a permutation null would bridge 13.8±3.3 and 77.9±8.5 pairs (P<0.001). Third, zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is commingled with generic drift. An exploratory θ^* allocation attains the largest full-pipeline gain (+4.36% vs. +0.80%/+0.64%,pp) but is marginal under Welch test.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
46 of 55 references.
References · 46
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
34 more
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
- 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2023Data Selection for Language Models via Importance Resamplingcited by 322
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
- 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
- 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
- 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
- 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
- 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
- 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
- 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
- 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
- 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
- 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
- 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
- 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
- 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
- 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
- 2025CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Modelscited by 19
- 2025A Survey on LLM Mid-Trainingcited by 18
- 2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 11
- 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
- 2025Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoningcited by 6
- 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
- 2025LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuningcited by 1
Built on by
Not cited yet.