mlsanityData to 14 Oct 2026

Accelerating Vision Transformers with Adaptive Patch Sizes

arXiv 2510.18091 ↗First posted 20 Oct 2025cs.CV cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and can be applied to a previously fine-tuned ViT, converging in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30% faster training and inference in visual QA, object detection, and semantic segmentation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 70 of its 80 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

70 of 80 references.

References · 70

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  12. 2017Random Erasing Data Augmentationcited by 1,239
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  15. 2022Token Merging: Your ViT But Fastercited by 1,021
  16. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  17. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  18. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  19. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  20. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
  21. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  22. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  23. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  24. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  25. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  26. 2021Post-Training Quantization for Vision Transformercited by 278
  27. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  28. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  29. 2023Token Merging for Fast Stable Diffusioncited by 224
  30. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  31. 2022FlexiViT: One Model for All Patch Sizescited by 152
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
  34. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  35. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  36. 2017Compressed Video Action Recognitioncited by 106
  37. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 70
  38. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splattingcited by 11
  2. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 4
  3. 2026TrajTok: Learning Trajectory Tokens enables better Video Understandingcited by 3
  4. 2026DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunkingcited by 3
  5. 2025Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformercited by 2
  6. 2026DVSM: Decoder-only View Synthesis Model Done Rightcited by 2
  7. 2026Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reductioncited by 0
  8. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  9. 2026MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generationcited by 0
  10. 2026PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inferencecited by 0
  11. 2026Variable-Granularity Tokenization for High-Resolution Object Detectioncited by 0
  12. 2026Dynamic Quadtree Tokenization and Transformer for Adaptive Mesh PDE Forecastingcited by 0
1 more
  1. 2026VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMscited by 0