mlsanityData to 14 Oct 2026

Forgetting of task-specific knowledge in model merging-based continual learning

arXiv 2507.23311 ↗First posted 31 Jul 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates the linear merging of models in the context of continual learning (CL). Using controlled visual cues in computer vision experiments, we demonstrate that merging largely preserves or enhances shared knowledge, while unshared task-specific knowledge rapidly degrades. We further find that merging models from an incremental training process consistently outperforms merging models trained in parallel.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

31 of 36 references.

References · 31

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
19 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  2. 2020What is being transferred in transfer learning?cited by 368
  3. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  5. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  6. 2022Patching open-vocabulary models by interpolating weightscited by 190
  7. 2014Qualitatively characterizing neural network optimization problemscited by 188
  8. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  9. 2023Deep Model Fusion: A Surveycited by 91
  10. 2024Continual Learning and Catastrophic Forgettingcited by 77
  11. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  12. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 42
  13. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  14. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  15. 2023Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also Howcited by 16
  16. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  17. 2024Continual Learning with Weight Interpolationcited by 13
  18. 2024Where is the Truth? The Risk of Getting Confounded in a Continual Worldcited by 8
  19. 2023Chunking: Continual Learning is not just about Distribution Shiftcited by 3

Built on by

Not cited yet.