# Activation-Informed Merging of Large Language Models

arXiv: https://arxiv.org/abs/2502.02421 · HTML: https://mlsanity.com/paper/2502.02421/

## Key facts

- Cited by 29 papers in the map (data through 7 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 30 of 36)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization](https://mlsanity.com/paper/2212.10445/) (2022)
- [Model Stock: All we need is just a few fine-tuned models](https://mlsanity.com/paper/2403.19522/) (2024)

## Built on by (most cited first, 29 of 29)

- [Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging](https://mlsanity.com/paper/2503.20641/) (2025)
- [CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation](https://mlsanity.com/paper/2505.14646/) (2025)
- [Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Models](https://mlsanity.com/paper/2508.02120/) (2025)
- [Activation-Guided Consensus Merging for Large Language Models](https://mlsanity.com/paper/2505.14009/) (2025)
- [Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA](https://mlsanity.com/paper/2512.17910/) (2025)
- [RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format](https://mlsanity.com/paper/2602.22538/) (2026)
- [Merge-of-Thought Distillation](https://mlsanity.com/paper/2509.08814/) (2025)
- [RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior](https://mlsanity.com/paper/2508.03140/) (2025)
- [Robust Policy Optimization to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/2602.08813/) (2026)
- [Label-Free Cross-Task LoRA Merging with Null-Space Compression](https://mlsanity.com/paper/2603.26317/) (2026)
- [TiMoE: Time-Aware Mixture of Language Experts](https://mlsanity.com/paper/2508.08827/) (2025)
- [ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Merging](https://mlsanity.com/paper/2601.07309/) (2026)
- [Can Heterogeneous Language Models Be Fused?](https://mlsanity.com/paper/2604.01674/) (2026)
- [Weight Patching: Toward Source-Level Mechanistic Localization in LLMs](https://mlsanity.com/paper/2604.13694/) (2026)
- [CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics](https://mlsanity.com/paper/2605.09584/) (2026)
- [Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging](https://mlsanity.com/paper/2605.29489/) (2026)
- [Personalized Collaborative Learning with Affinity-Based Variance Reduction](https://mlsanity.com/paper/2510.16232/) (2025)
- [QA-Merging: Query-Adaptive Reasoning via Layer Selective Model Merging](https://mlsanity.com/paper/2601.03506/) (2026)
- [Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations](https://mlsanity.com/paper/2602.03237/) (2026)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [Unlocking the Potential of Continual Model Merging: An ODE Perspective](https://mlsanity.com/paper/2605.19409/) (2026)
- [F-TIS: Harnessing Diverse Models in Collaborative GRPO](https://mlsanity.com/paper/2605.22537/) (2026)
- [RogueMerge: Robust and Unified Attacks against LLM Model Merging](https://mlsanity.com/paper/2606.03344/) (2026)
- [When Model Merging Breaks Routing: Training-Free Calibration for MoE](https://mlsanity.com/paper/2606.03391/) (2026)
- [AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention](https://mlsanity.com/paper/2608.11758/) (2026)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)
- [Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs](https://mlsanity.com/paper/2603.21705/) (2026)
- [CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing](https://mlsanity.com/paper/2605.14084/) (2026)
- [Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Merging](https://mlsanity.com/paper/2608.10447/) (2026)

## Abstract (arXiv)

Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance.