mlsanityData to 7 Oct 2026

Activation-Informed Merging of Large Language Models

arXiv 2502.02421 ↗First posted 4 Feb 2025cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining computational efficiency. This paper introduces Activation-Informed Merging (AIM), a technique that integrates the information from the activation space of LLMs into the merging process to improve performance and robustness. AIM is designed as a flexible, complementary solution that is applicable to any existing merging method. It aims to preserve critical weights from the base model, drawing on principles from continual learning (CL) and model compression. Utilizing a task-agnostic calibration set, AIM selectively prioritizes essential weights during merging. We empirically demonstrate that AIM significantly enhances the performance of merged models across multiple benchmarks. Our findings suggest that considering the activation-space information can provide substantial advancements in the model merging strategies for LLMs, with up to a 40% increase in benchmark performance.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

36 of 45 references.

References · 36

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2022Editing Models with Task Arithmeticcited by 1,076
24 more
  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  7. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  9. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 225
  13. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  14. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  15. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  16. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  17. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  18. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  19. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  20. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  21. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  22. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  23. 2022ORFit: One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squarescited by 12
  24. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 20.

29 papers.

Sort:

By area:

  1. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  2. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 39
  3. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  4. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  5. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRAcited by 7
  6. 2026RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Formatcited by 7
  7. 2025Merge-of-Thought Distillationcited by 6
  8. 2025RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Priorcited by 5
  9. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  10. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 3
  11. 2025TiMoE: Time-Aware Mixture of Language Expertscited by 2
  12. 2026ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Mergingcited by 2
17 more
  1. 2026Can Heterogeneous Language Models Be Fused?cited by 2
  2. 2026Weight Patching: Toward Source-Level Mechanistic Localization in LLMscited by 2
  3. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 2
  4. 2026Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Mergingcited by 2
  5. 2025Personalized Collaborative Learning with Affinity-Based Variance Reductioncited by 1
  6. 2026QA-Merging: Query-Adaptive Reasoning via Layer Selective Model Mergingcited by 1
  7. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  8. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 1
  9. 2026Unlocking the Potential of Continual Model Merging: An ODE Perspectivecited by 1
  10. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPOcited by 1
  11. 2026RogueMerge: Robust and Unified Attacks against LLM Model Mergingcited by 1
  12. 2026When Model Merging Breaks Routing: Training-Free Calibration for MoEcited by 1
  13. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtentioncited by 1
  14. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  15. 2026Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMscited by 0
  16. 2026CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editingcited by 0
  17. 2026Towards Efficient Reasoning in LLM-Based Recommender Systems via Model Mergingcited by 0