mlsanityData to 14 Oct 2026

Model Merging in Pre-training of Large Language Models

arXiv 2505.12082 ↗First posted 17 May 2025cs.CL cs.LGcited by 37

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 54 of its 65 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

54 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Qwen2.5 Technical Report19 Dec
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  13. 2022Editing Models with Task Arithmeticcited by 1,120
  14. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  15. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  16. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  17. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  18. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  19. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  20. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  21. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  22. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  23. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  24. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  25. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  26. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  27. 2018An Empirical Model of Large-Batch Trainingcited by 281
  28. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  29. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  30. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  31. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  32. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  33. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  34. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60
  35. 2024What Matters for Model Merging at Scale?cited by 55
  36. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  37. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  38. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 19.

37 papers.

Sort:

By area:

  1. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  2. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  3. 2025Kwai Keye-VL Technical Reportcited by 47
  4. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  5. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  6. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  7. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  8. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  9. 2025Merge-of-Thought Distillationcited by 7
  10. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
  11. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  12. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 5
25 more
  1. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  2. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  3. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  4. 2025Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Financecited by 2
  5. 2025Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Mergingcited by 2
  6. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
  7. 2025MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamicscited by 1
  8. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  9. 2026Optimizing Visual Generative Models via Distribution-wise Rewardscited by 1
  10. 2026Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Trainingcited by 1
  11. 2025ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Modelscited by 0
  12. 2025Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Modelscited by 0
  13. 2025MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Modelscited by 0
  14. 2025JT-Safe: Intrinsically Enhancing the Safety and Trustworthiness of LLMscited by 0
  15. 2026CoPeP: Benchmarking Continual Pretraining for Protein Language Modelscited by 0
  16. 2026Mi:dm K 2.5 Procited by 0
  17. 2026LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learningcited by 0
  18. 2026Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Expertscited by 0
  19. 2026Uncensored Open-weight Models: Redistribution as the Persistence Layercited by 0
  20. 2026Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stackcited by 0
  21. 2026Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMscited by 0
  22. 2026QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistencycited by 0
  23. 2026IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligencecited by 0
  24. 2026Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectoriescited by 0
  25. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  2. 2024Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
  3. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
  4. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
  5. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement
  6. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
  7. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  8. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking