mlsanityData to 14 Oct 2026

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv 2609.08966 ↗First posted 8 Sep 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

37 of 41 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
25 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  13. 2025Generalizing Verifiable Instruction Followingcited by 194
  14. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  15. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  16. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  17. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  18. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  19. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  20. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  21. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 28
  22. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  23. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  24. 2026Arcee Trinity Large Technical Reportcited by 11
  25. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
  2. 2024Sparse Checkpointing for Fast and Reliable MoE Training
  3. 2025Uncertainty-Guided Checkpoint Selection for Reinforcement Finetuning of Large Language Models
  4. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
  5. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  6. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  7. 2025Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging
  8. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data