mlsanityData to 7 Oct 2026

FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer

arXiv 2212.03145 ↗First posted 6 Dec 2022cs.CVcited by 133

Not on the map. Why?

Abstract (arXiv)

Recent work has explored the potential to adapt a pre-trained vision transformer (ViT) by updating only a few parameters so as to improve storage efficiency, called parameter-efficient transfer learning (PETL). Current PETL methods have shown that by tuning only 0.5% of the parameters, ViT can be adapted to downstream tasks with even better performance than full fine-tuning. In this paper, we aim to further promote the efficiency of PETL to meet the extreme storage constraint in real-world applications. To this end, we propose a tensorization-decomposition framework to store the weight increments, in which the weights of each ViT are tensorized into a single 3D tensor, and their increments are then decomposed into lightweight factors. In the fine-tuning process, only the factors need to be updated and stored, termed Factor-Tuning (FacT). On VTAB-1K benchmark, our method performs on par with NOAH, the state-of-the-art PETL method, while being 5x more parameter-efficient. We also present a tiny version that only uses 8K (0.01% of ViT's parameters) trainable parameters but outperforms full fine-tuning and many other PETL methods such as VPT and BitFit. In few-shot settings, FacT also beats all PETL baselines using the fewest parameters, demonstrating its strong capability in the low-data regime.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

35 of 56 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
23 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2017Learning multiple visual domains with residual adapterscited by 553
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  11. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  13. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  14. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  15. 2022Neural Prompt Searchcited by 133
  16. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  17. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  18. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  19. 2019A Tensorized Transformer for Language Modelingcited by 85
  20. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  21. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  22. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  23. 2019On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compressioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 57; 2025: 32; 2026: 18.

133 papers.

Sort:

By area:

  1. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  2. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  3. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  4. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  5. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  6. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  7. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  8. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  9. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  10. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  11. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  12. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
38 more
  1. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  2. 2023Visual Tuningcited by 34
  3. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  4. 2024Visual Fourier Prompt Tuningcited by 30
  5. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 29
  6. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  7. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  8. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  9. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  10. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  11. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  12. 2023SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstructioncited by 23
  13. 2023DTL: Disentangled Transfer Learning for Visual Recognitioncited by 22
  14. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  15. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  16. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21
  17. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  18. 2024Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuningcited by 20
  19. 2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approachcited by 18
  20. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  21. 2023Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuningcited by 17
  22. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  23. 2024Adapters Strike Backcited by 16
  24. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  25. 2023Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Modelcited by 15
  26. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  27. 2025GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Modelcited by 13
  28. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  29. 2024Enhancing Diagnostic Accuracy in Rare and Common Fundus Diseases with a Knowledge-Rich Vision-Language Modelcited by 12
  30. 2024Large Convolutional Model Tuning via Filter Subspacecited by 11
  31. 2024M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehensioncited by 11
  32. 2024SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learningcited by 11
  33. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  34. 2024SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modulescited by 10
  35. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  36. 2024IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Modelscited by 9
  37. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  38. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9

50 most cited of 133 All 133 (JSON)