mlsanityData to 7 Oct 2026

Overcoming Generic Knowledge Loss with Selective Parameter Update

arXiv 2308.12462 ↗First posted 23 Aug 2023cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Foundation models encompass an extensive knowledge base and offer remarkable transferability. However, this knowledge becomes outdated or insufficient over time. The challenge lies in continuously updating foundation models to accommodate novel information while retaining their original capabilities. Leveraging the fact that foundation models have initial knowledge on various tasks and domains, we propose a novel approach that, instead of updating all parameters equally, localizes the updates to a sparse set of parameters relevant to the task being learned. We strike a balance between efficiency and new task performance, while maintaining the transferability and generalizability of foundation models. We extensively evaluate our method on foundational vision-language models with a diverse spectrum of continual learning tasks. Our method achieves improvements on the accuracy of the newly learned tasks up to 7% while preserving the pretraining knowledge with a negligible decrease of 0.9% on a representative control set accuracy.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

49 of 67 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2016Learning without Forgettingcited by 2,465
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
37 more
  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  8. 2022Mass-Editing Memory in a Transformercited by 976
  9. 2021Learning to Prompt for Continual Learningcited by 938
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  13. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  14. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  15. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  16. 2021Fast Model Editing at Scalecited by 491
  17. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  18. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  19. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  20. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  21. 2020Supermasks in Superpositioncited by 223
  22. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  23. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  24. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  25. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  26. 2020Adversarial Continual Learningcited by 140
  27. 2022Analyzing Transformers in Embedding Spacecited by 131
  28. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  29. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  30. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  31. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  32. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  33. 2021Editing a classifier by rewriting its prediction rulescited by 64
  34. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  35. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  36. 2023Parameter-Level Soft-Masking for Continual Learningcited by 39
  37. 2023Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learningcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  4. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  5. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  6. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  7. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learningcited by 6
  8. 2025Learning without Isolation: Pathway Protection for Continual Learningcited by 6
  9. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  10. 2024Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retentioncited by 5
  11. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  12. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 4
11 more
  1. 2024Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inferencecited by 3
  2. 2024Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learningcited by 2
  3. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  4. 2025Efficient Few-Shot Continual Learning in Vision-Language Modelscited by 2
  5. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  6. 2024Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Modelscited by 1
  7. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
  8. 2024Lifelong Person Searchcited by 0
  9. 2025Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selectioncited by 0
  10. 2025Closing the Oracle Gap: Increment Vector Transformation for Class Incremental Learningcited by 0
  11. 2026Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameter Importance-Driven Continual Learning for Foundation Models
  2. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  3. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
  4. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Survey
  5. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  6. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  7. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  8. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models