# Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2404.17245 · HTML: https://mlsanity.com/paper/2404.17245/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning](https://mlsanity.com/paper/2210.17451/) (2022)
- [Investigating the Catastrophic Forgetting in Multimodal Large Language Models](https://mlsanity.com/paper/2309.10313/) (2023)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [Parameter-efficient Model Adaptation for Vision Transformers](https://mlsanity.com/paper/2203.16329/) (2022)
- [Continual Adaptation of Visual Representations via Domain Randomization and Meta-learning](https://mlsanity.com/paper/2012.04324/) (2020)
- [Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy](https://mlsanity.com/paper/2307.16867/) (2023)
- [Optimizing Relevance Maps of Vision Transformers Improves Robustness](https://mlsanity.com/paper/2206.01161/) (2022)
- [Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning](https://mlsanity.com/paper/2210.01035/) (2022)
- [Parameter-efficient Tuning of Large-scale Multimodal Foundation Model](https://mlsanity.com/paper/2305.08381/) (2023)
- [Entropic Open-set Active Learning](https://mlsanity.com/paper/2312.14126/) (2023)
- [Transfer Learning for Fine-grained Classification Using Semi-supervised Learning and Visual Transformers](https://mlsanity.com/paper/2305.10018/) (2023)
- [Open-Set Automatic Target Recognition](https://mlsanity.com/paper/2211.05883/) (2022)

## Built on by (most cited first, 17 of 17)

- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [Interpretable Image Classification via Non-parametric Part Prototype Learning](https://mlsanity.com/paper/2503.10247/) (2025)
- [Fine-tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models](https://mlsanity.com/paper/2504.17397/) (2025)
- [Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation](https://mlsanity.com/paper/2501.15377/) (2025)
- [SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning](https://mlsanity.com/paper/2405.09394/) (2024)
- [MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations](https://mlsanity.com/paper/2406.12368/) (2024)
- [Norm-Bounded Low-Rank Adaptation](https://mlsanity.com/paper/2501.19050/) (2025)
- [Enhancing knowledge retention for continual learning with domain-specific adapters and features gating](https://mlsanity.com/paper/2504.08613/) (2025)
- [Elytra: A Flexible Framework for Securing Large Vision Systems](https://mlsanity.com/paper/2506.00661/) (2025)
- [Optimisation Is Not What You Need](https://mlsanity.com/paper/2507.03045/) (2025)
- [Reducing Simulation Dependence in Neutrino Telescopes with Masked Point Transformers](https://mlsanity.com/paper/2510.01733/) (2025)
- [Face, Whole-Person, and Object Classification in a Unified Space Via The Interleaved Multi-Domain Identity Curriculum](https://mlsanity.com/paper/2511.19846/) (2025)
- [Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures](https://mlsanity.com/paper/2605.19478/) (2026)
- [Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification](https://mlsanity.com/paper/2606.27582/) (2026)
- [Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology](https://mlsanity.com/paper/2609.17868/) (2026)
- [Video-STLayout Pre-training](https://mlsanity.com/paper/2609.24031/) (2026)
- [A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning](https://mlsanity.com/paper/2609.33424/) (2026)

## Abstract (arXiv)

Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-trained on ImageNet-1k loses over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100. Overcoming this stability-plasticity dilemma is crucial for enabling ViTs to continuously learn and adapt to new domains while preserving their initial knowledge. In this work, we study two new parameter-efficient fine-tuning strategies: (1)~Block Expansion, and (2) Low-rank adaptation (LoRA). Our experiments reveal that using either Block Expansion or LoRA on self-supervised pre-trained ViTs surpass fully fine-tuned ViTs in new domains while offering significantly greater parameter efficiency. Notably, we find that Block Expansion experiences only a minimal performance drop in the pre-training domain, thereby effectively mitigating catastrophic forgetting in pre-trained ViTs.