# Checkpoint Merging via Bayesian Optimization in LLM Pretraining

arXiv: https://arxiv.org/abs/2403.19390 · HTML: https://mlsanity.com/paper/2403.19390/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [ELLE: Efficient Lifelong Pre-training for Emerging Data](https://mlsanity.com/paper/2203.06311/) (2022)
- [Learning to select data for transfer learning with Bayesian Optimization](https://mlsanity.com/paper/1707.05246/) (2017)
- [Token Dropping for Efficient BERT Pretraining](https://mlsanity.com/paper/2203.13240/) (2022)
- [Early Weight Averaging meets High Learning Rates for LLM Pre-training](https://mlsanity.com/paper/2306.03241/) (2023)
- [Checkpoint Ensembles: Ensemble Methods from a Single Training Process](https://mlsanity.com/paper/1710.03282/) (2017)

## Built on by (most cited first, 27 of 27)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [Model Merging in Pre-training of Large Language Models](https://mlsanity.com/paper/2505.12082/) (2025)
- [RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness](https://mlsanity.com/paper/2502.17159/) (2025)
- [WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training](https://mlsanity.com/paper/2507.17634/) (2025)
- [Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging](https://mlsanity.com/paper/2406.16330/) (2024)
- [Bayesian Concept Bottleneck Models with LLM Priors](https://mlsanity.com/paper/2410.15555/) (2024)
- [Dynamic Fisher-weighted Model Merging via Bayesian Optimization](https://mlsanity.com/paper/2504.18992/) (2025)
- [Scalable Model Merging with Progressive Layer-wise Distillation](https://mlsanity.com/paper/2502.12706/) (2025)
- [Ensemble Learning for Large Language Models in Text and Code Generation: A Survey](https://mlsanity.com/paper/2503.13505/) (2025)
- [Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection](https://mlsanity.com/paper/2504.13938/) (2025)
- [LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging](https://mlsanity.com/paper/2502.10749/) (2025)
- [Why Do More Experts Fail? A Theoretical Analysis of Model Merging](https://mlsanity.com/paper/2505.21226/) (2025)
- [Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging](https://mlsanity.com/paper/2502.12217/) (2025)
- [Model Assembly Learning with Heterogeneous Layer Weight Merging](https://mlsanity.com/paper/2503.21657/) (2025)
- [Learning Explainable Dense Reward Shapes via Bayesian Optimization](https://mlsanity.com/paper/2504.16272/) (2025)
- [Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?](https://mlsanity.com/paper/2510.14387/) (2025)
- [EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusion](https://mlsanity.com/paper/2512.04545/) (2025)
- [Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training](https://mlsanity.com/paper/2605.26484/) (2026)
- [Extrapolation Merging: Keep Improving With Extrapolation and Merging](https://mlsanity.com/paper/2503.04834/) (2025)
- [Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging](https://mlsanity.com/paper/2504.18580/) (2025)
- [MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics](https://mlsanity.com/paper/2510.09295/) (2025)
- [Optimizing Visual Generative Models via Distribution-wise Rewards](https://mlsanity.com/paper/2607.02291/) (2026)
- [FedAvgen: Metadata for Model Aggregation In Communication Systems](https://mlsanity.com/paper/2505.05486/) (2025)
- [Composable Cross-prompt Essay Scoring by Merging Models](https://mlsanity.com/paper/2505.18548/) (2025)
- [ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services](https://mlsanity.com/paper/2606.02606/) (2026)
- [Multi-Objective Bayesian Optimization for Model Merging](https://mlsanity.com/paper/2608.14264/) (2026)

## Abstract (arXiv)

The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. To alleviate this issue, we propose checkpoint merging in pretraining LLM. This method utilizes LLM checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate that: (1) Our proposed methodology exhibits the capacity to augment pretraining, presenting an opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.