mlsanityData to 7 Oct 2026

Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

arXiv 2403.11808 ↗First posted 18 Mar 2024cs.CVcited by 25

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced adapter to further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 84 of its 101 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

84 of 101 references.

References · 84

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2016Layer Normalizationcited by 5,579
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  14. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  15. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  16. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  17. 2022Token Merging: Your ViT But Fastercited by 995
  18. 2021Scaling Vision Transformerscited by 993
  19. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  20. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  21. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  22. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  23. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  24. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  25. 2020A Survey on Visual Transformercited by 653
  26. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  27. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  28. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  29. 2021Dynamic Neural Networks: A Surveycited by 405
  30. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  31. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  32. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  33. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  34. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  35. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  36. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  37. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  38. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  2. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  3. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  4. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  5. 2025Recurrent Diffusion for Large-Scale Parameter Generationcited by 15
  6. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  7. 2024SparseDiT: Token Sparsification for Efficient Diffusion Transformercited by 9
  8. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  9. 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformercited by 6
  10. 2025Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learningcited by 5
  11. 2025EA-ViT: Efficient Adaptation for Elastic Vision Transformercited by 5
  12. 2025Neural-Driven Image Editingcited by 4
13 more
  1. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  2. 2024Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformercited by 2
  3. 2026FARTrack: Fast Autoregressive Visual Tracking with High Performancecited by 2
  4. 2025Dynamic Rank Adaptation for Vision-Language Modelscited by 0
  5. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0
  6. 2025Decoupled Multi-Predictor Optimization for Inference-Efficient Model Tuningcited by 0
  7. 2025Mechanisms of Non-Monotonic Scaling in Vision Transformerscited by 0
  8. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  9. 2026Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuningcited by 0
  10. 2026Spike-NVPT: Learning Robust Visual Prompts via Bio-Inspired Temporal Filtering and Discretizationcited by 0
  11. 2026Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?cited by 0
  12. 2026Putting Registers to Work: Task Registers for Token Pruning in Vision Transformerscited by 0
  13. 2026Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation
  2. 2023Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation
  3. 2026Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
  4. 2026Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing
  5. 2026Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
  6. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models
  7. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models
  8. 2026ViT-5: Vision Transformers for The Mid-2020s