mlsanityData to 14 Oct 2026

APLA: A Simple Adaptation Method for Vision Transformers

arXiv 2503.11335 ↗First posted 14 Mar 2025cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at https://github.com/MoeinSorkhei/APLA.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 49 of its 70 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

49 of 70 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Layer Normalizationcited by 5,669
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
37 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2022Visual Prompt Tuningcited by 1,681
  5. 2019Natural Adversarial Examplescited by 1,408
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  9. 2014How transferable are features in deep neural networks?cited by 1,225
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  13. 2021Going deeper with Image Transformerscited by 764
  14. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  16. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  17. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  18. 2020What is being transferred in transfer learning?cited by 368
  19. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  20. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  21. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  22. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
  23. 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
  24. 2022Neural Prompt Searchcited by 135
  25. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  26. 2021The Efficiency Misnomercited by 98
  27. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  28. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  29. 2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
  30. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  31. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  32. 2023Efficient Adaptation of Large Vision Transformer via Adapter Re-Composingcited by 35
  33. 2023Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptationcited by 18
  34. 2024Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approachcited by 18
  35. 2024Adapters Strike Backcited by 16
  36. 2023Pretrained ViTs Yield Versatile Representations For Medical Imagescited by 13
  37. 2024Learning from Offline Foundation Features with Tensor Augmentationscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Efficient Self-Supervised Adaptation for Medical Image Analysiscited by 0