mlsanityData to 14 Oct 2026

Adapting Vision Foundation Models with Cascaded Semantics

arXiv 2608.05393 ↗First posted 5 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Prompt tuning, a leading parameter-efficient adaptation paradigm in NLP, has recently been extended to computer vision. Visual prompt tuning (VPT) adapts pre-trained vision transformers (ViTs) by updating a small set of additional prompt parameters. However, existing visual prompts are randomly initialized and do not exploit prior knowledge, such as instructions in NLP. We address this gap by injecting two complementary semantic priors into VPT. Fundamental image priors, including color, texture, and shape, are extracted with classical hand-crafted operators and injected into the input space, while self-attention maps provide instance-aware semantics in the feature space. We further propose a cascaded scheme that integrates both priors throughout ViT adaptation. Experiments on 34 challenging image classification datasets demonstrate superior downstream adaptation while tuning only 0.74% of ViT parameters. Project page: https://xixiaouab.github.io/Cascaded-Semantics/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 83 of its 121 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

83 of 121 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  3. 2013Describing Textures in the Wildcited by 2,125
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2014How transferable are features in deep neural networks?cited by 1,225
  11. 2021Transformers in Vision: A Surveycited by 1,013
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  13. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  14. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  15. 2018Image Transformercited by 772
  16. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  17. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  18. 2018MINE: Mutual Information Neural Estimationcited by 683
  19. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  20. 2020A Survey on Visual Transformercited by 654
  21. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  22. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 608
  23. 2018Do Better ImageNet Models Transfer Better?cited by 565
  24. 2017Learning multiple visual domains with residual adapterscited by 557
  25. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  26. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  27. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  28. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  29. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  30. 2015Training Very Deep Networkscited by 173
  31. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 121
  32. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  33. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  34. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  35. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 69
  36. 2022Visual Recognition with Deep Nearest Centroidscited by 59
  37. 2023Diversity-Aware Meta Visual Promptingcited by 58
  38. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57

50 most cited of 83 All 83 (JSON)

Built on by

Not cited yet.