# MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation

arXiv: https://arxiv.org/abs/2406.07529 · HTML: https://mlsanity.com/paper/2406.07529/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 86 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)

## Built on by (most cited first, 20 of 20)

- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent](https://mlsanity.com/paper/2501.01230/) (2025)
- [RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness](https://mlsanity.com/paper/2502.17159/) (2025)
- [It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization](https://mlsanity.com/paper/2407.00487/) (2024)
- [HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models](https://mlsanity.com/paper/2409.18893/) (2024)
- [A Systematic Study of In-the-Wild Model Merging for Large Language Models](https://mlsanity.com/paper/2511.21437/) (2025)
- [COPA: Comparing the incomparable in multi-objective model evaluation](https://mlsanity.com/paper/2503.14321/) (2025)
- [Discovering Physical Directions in Weight Space: Composing Neural PDE Experts](https://mlsanity.com/paper/2605.14546/) (2026)
- [Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging](https://mlsanity.com/paper/2512.01461/) (2025)
- [PACT: Preserving Anchored Cores in Task-vectors for Model Merging](https://mlsanity.com/paper/2606.18627/) (2026)
- [Variational Model Merging for Pareto Front Estimation in Multitask Finetuning](https://mlsanity.com/paper/2412.08147/) (2024)
- [StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation](https://mlsanity.com/paper/2506.04567/) (2025)
- [DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging](https://mlsanity.com/paper/2508.01148/) (2025)
- [Building Multi-Task Agentic LLMs via Two-Phase Distillation](https://mlsanity.com/paper/2606.30044/) (2026)
- [Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback](https://mlsanity.com/paper/2609.13714/) (2026)
- [On the Vulnerability of Parameter-Level Defenses to Model Merging](https://mlsanity.com/paper/2606.30360/) (2026)
- [Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space](https://mlsanity.com/paper/2607.01689/) (2026)
- [Multi-Objective Bayesian Optimization for Model Merging](https://mlsanity.com/paper/2608.14264/) (2026)
- [The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization](https://mlsanity.com/paper/2609.39055/) (2026)
- [SLIM: Simplex-Lattice Interpolation Merging](https://mlsanity.com/paper/2610.01037/) (2026)

## Abstract (arXiv)

Model merging has emerged as an effective approach to combine multiple single-task models into a multitask model. This process typically involves computing a weighted average of the model parameters without any additional training. Existing model-merging methods focus on enhancing average task accuracy. However, interference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a set of solutions with various trade-offs can be more informative, helping practitioners make decisions based on diverse preferences. In this paper, we introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from a pre-selected set of scaling coefficients. Experimental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.