mlsanityData to 7 Oct 2026

Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

arXiv 2609.09081 ↗First posted 8 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Mid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it. In a controlled logical-reasoning setting (Qwen3-8B-Base, with a 4B replication; five semantically rule-disjoint KOR-Bench domains) we train 30 allocations spanning the five-domain simplex, 24 sweep configurations plus six withheld from the fit, at five seeds each. Three findings emerge. First, every domain has an interior coverage optimum: the moderate band (10%-40%) is best for all five domains, and a calibrated permutation test for quadratic interiority gives P≈0.010; the fitted mid-training-only curves, with 8B peaks between 9.9% and 35.1%, reproduce for curve shape but not peak location. Second, the gaps survive a fixed-budget alignment pass: compensatory SFT raises 116/120 cells (mean +4.32%) yet bridges 0/240 pairs at a 5% threshold and 30/240 at a 10% ratio, an equal-budget uniform control behaves almost identically, and a permutation null would bridge 13.8±3.3 and 77.9±8.5 pairs (P<0.001). Third, zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is commingled with generic drift. An exploratory θ^* allocation attains the largest full-pipeline gain (+4.36% vs. +0.80%/+0.64%,pp) but is marginal under Welch test.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

46 of 55 references.

References · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
34 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  9. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  11. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  12. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  13. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  14. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  15. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  16. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  17. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  18. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  19. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  20. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  21. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  22. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  23. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  24. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  25. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
  26. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  27. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  28. 2025CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Modelscited by 19
  29. 2025A Survey on LLM Mid-Trainingcited by 18
  30. 2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 11
  31. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  32. 2025Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoningcited by 6
  33. 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
  34. 2025LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuningcited by 1

Built on by

Not cited yet.