mlsanityData to 7 Oct 2026

Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation

arXiv 2501.15377 ↗First posted 26 Jan 2025cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Adapting deep learning models to new domains often requires computationally intensive retraining and risks catastrophic forgetting. While fine-tuning enables domain-specific adaptation, it can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance. Pre-trained zero-shot models like CLIP offer strong generalization but may suffer degraded robustness after fine-tuning. Building on Task Adaptive Parameter Sharing (TAPS), we propose a simple yet effective extension as a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks. Our approach minimizes knowledge loss, retains its generalization strengths under domain shifts, and significantly reduces computational costs compared to traditional fine-tuning. We demonstrate that effective fine-tuning can be achieved with as few as 5% of active blocks, substantially improving efficiency. Evaluations on pre-trained models such as CLIP and DINO-ViT demonstrate our method's broad applicability and effectiveness in maintaining performance and knowledge retention.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

52 of 67 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  5. 2022Visual Prompt Tuningcited by 1,674
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2021Robust fine-tuning of zero-shot modelscited by 785
  13. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  14. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  15. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  17. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  18. 2024LoRA Learns Less and Forgets Lesscited by 313
  19. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  20. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  21. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  22. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  23. 2016Learning Features by Watching Objects Movecited by 138
  24. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  25. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  26. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  27. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  28. 2024Multi-LoRA Composition for Image Generationcited by 80
  29. 2024Continual Learning and Catastrophic Forgettingcited by 73
  30. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  31. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  32. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  33. 2022Task Adaptive Parameter Sharing for Multi-Task Learningcited by 42
  34. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  35. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  36. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  37. 2021A Practical Contrastive Learning Framework for Single-Image Super-Resolutioncited by 30
  38. 2024Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgettingcited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Fine-tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Modelscited by 9
  2. 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generationcited by 2
  3. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  4. 2025Joint Flashback Adaptation for Forgetting-Resistant Instruction Tuningcited by 0
  5. 2025mmHSense: Multi-Modal and Distributed mmWave ISAC Datasets for Human Sensingcited by 0
  6. 2026Transferable Optimization Network for Cross-Domain Image Reconstructioncited by 0