mlsanityData to 7 Oct 2026

MagMax: Leveraging Model Merging for Seamless Continual Learning

arXiv 2407.06322 ↗First posted 8 Jul 2024cs.LG cs.AI cs.CVcited by 67

Not on the map. Why?

Abstract (arXiv)

This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge. Distinct from traditional continual learning methods that aim to reduce forgetting during task training, MagMax combines sequential fine-tuning with a maximum magnitude weight selection for effective knowledge integration across tasks. Our initial contribution is an extensive examination of model merging techniques, revealing that simple approaches like weight averaging and random weight selection surprisingly hold up well in various continual learning contexts. More importantly, we present MagMax, a novel model-merging strategy that enables continual learning of large pre-trained models for successive tasks. Our thorough evaluation demonstrates the superiority of MagMax in various scenarios, including class- and domain-incremental learning settings. The code is available at this URL: https://github.com/danielm1405/magmax.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

42 of 55 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Segment Anythingcited by 8,158
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2016Learning without Forgettingcited by 2,465
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
30 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2022Editing Models with Task Arithmeticcited by 1,076
  6. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  7. 2021Learning to Prompt for Continual Learningcited by 938
  8. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  15. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  16. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  17. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  18. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  19. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  20. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  21. 2019Model Fusion via Optimal Transportcited by 217
  22. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  23. 2022Patching open-vocabulary models by interpolating weightscited by 187
  24. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  25. 2022FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learningcited by 120
  26. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  27. 2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 55
  28. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  29. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  30. 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learningcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 31; 2026: 26.

67 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  3. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  4. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  5. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  6. 2024Revisiting Weight Averaging for Model Mergingcited by 24
  7. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  8. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  9. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  10. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  11. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  12. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
38 more
  1. 2024NegMerge: Sign-Consensual Weight Merging for Machine Unlearningcited by 9
  2. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 9
  3. 2025Subspace-Boosted Model Mergingcited by 9
  4. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  5. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  6. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  7. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 7
  8. 2025Task Vector Quantization for Memory-Efficient Model Mergingcited by 5
  9. 2025Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learningcited by 5
  10. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  11. 2025Null-Space Filtering for Data-Free Continual Model Merging: Preserving Stability, Promoting Plasticitycited by 4
  12. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  13. 2024Low-Rank Continual Personalization of Diffusion Modelscited by 3
  14. 2024Sparse Orthogonal Parameters Tuning for Continual Learningcited by 2
  15. 2024MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Mergingcited by 2
  16. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  17. 2025MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosiscited by 2
  18. 2025Toward a Holistic Approach to Continual Model Mergingcited by 2
  19. 2025MAGIC: Achieving Superior Model Merging via Magnitude Calibrationcited by 2
  20. 2026Bridging Domains through Subspace-Aware Model Mergingcited by 2
  21. 2024SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptationcited by 1
  22. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  23. 2025DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmeticcited by 1
  24. 2025GLAM: Efficient Continual Learning at Scale via Grouped LoRA Adapter Mergingcited by 1
  25. 2025Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defensescited by 1
  26. 2025Weight Weaving: Parameter Pooling for Data-Free Model Mergingcited by 1
  27. 2026Analytic Drift Resister for Non-Exemplar Continual Graph Learningcited by 1
  28. 2026DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editingcited by 1
  29. 2026ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Mergingcited by 1
  30. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  31. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1
  32. 2026Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrievalcited by 1
  33. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0
  34. 2025Med-LEGO: Editing and Adapting toward Generalist Medical Image Diagnosiscited by 0
  35. 2025Forgetting of task-specific knowledge in model merging-based continual learningcited by 0
  36. 2025Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgettingcited by 0
  37. 2025Learn Faster and Remember More: Balancing Exploration and Exploitation for Continual Test-time Adaptationcited by 0
  38. 2025Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learningcited by 0

50 most cited of 67 All 67 (JSON)