mlsanityData to 7 Oct 2026

Dynamic Grained Encoder for Vision Transformers

arXiv 2301.03831 ↗First posted 10 Jan 2023cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Transformers, the de-facto standard for language modeling, have been recently applied for vision tasks. This paper introduces sparse queries for vision transformers to exploit the intrinsic spatial redundancy of natural images and save computational costs. Specifically, we propose a Dynamic Grained Encoder for vision transformers, which can adaptively assign a suitable number of queries to each spatial region. Thus it achieves a fine-grained representation in discriminative regions while keeping high efficiency. Besides, the dynamic grained encoder is compatible with most vision transformer frameworks. Without bells and whistles, our encoder allows the state-of-the-art vision transformers to reduce computational complexity by 40%-60% while maintaining comparable performance on image classification. Extensive experiments on object detection and segmentation further demonstrate the generalizability of our approach. Code is available at https://github.com/StevenGrove/vtpack.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 50 of its 71 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

50 of 71 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  13. 2017Focal Loss for Dense Object Detectioncited by 749
  14. 2019Panoptic Feature Pyramid Networkscited by 644
  15. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  16. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  17. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  18. 2021Dynamic Neural Networks: A Surveycited by 405
  19. 2019Axial Attention in Multidimensional Transformerscited by 350
  20. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  21. 2018Context Encoding for Semantic Segmentationcited by 289
  22. 2020Sparse Sinkhorn Attentioncited by 250
  23. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  24. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  25. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  26. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  27. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  28. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  29. 2019Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networkscited by 118
  30. 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
  31. 2020Learning Dynamic Routing for Semantic Segmentationcited by 79
  32. 2020End-to-End Object Detection with Fully Convolutional Networkcited by 70
  33. 2020Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolationcited by 51
  34. 2019TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detectioncited by 36
  35. 2017Segmentation-Aware Convolutional Networks Using Local Attention Maskscited by 31
  36. 2019Learnable Tree Filter for Structure-preserving Feature Transformcited by 25
  37. 2020Fine-Grained Dynamic Head for Object Detectioncited by 16
  38. 2020Rethinking Learnable Tree Filter for Generic Feature Transformcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 7; 2025: 6; 2026: 2.

24 papers.

Sort:

By area:

  1. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  2. 2024Dynamic Diffusion Transformercited by 60
  3. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 59
  4. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  5. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  6. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  7. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  8. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  9. 2024GrootVL: Tree Topology is All You Need in State Space Modelcited by 13
  10. 2023BoxSnake: Polygonal Instance Segmentation with Box Supervisioncited by 12
  11. 2023Make A Long Image Short: Adaptive Token Length for Vision Transformerscited by 10
  12. 2023AiluRus: A Scalable ViT Framework for Dense Predictioncited by 10
12 more
  1. 2023AutoFocusFormer: Image Segmentation off the Gridcited by 9
  2. 2025SparseFormer: Detecting Objects in HRW Shots via Sparse Vision Transformercited by 7
  3. 2026Elastic Diffusion Transformercited by 7
  4. 2024Learning to Rank Patches for Unbiased Image Redundancy Reductioncited by 6
  5. 2025Dynamic Vision Mambacited by 6
  6. 2025Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learningcited by 5
  7. 2024TPC-ViT: Token Propagation Controller for Efficient Vision Transformercited by 2
  8. 2025Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splittingcited by 2
  9. 2024Successive Refinement in Large-Scale Computation: Advancing Model Inference Applicationscited by 0
  10. 2025Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentationcited by 0
  11. 2025MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mambacited by 0
  12. 2026Variable-Granularity Tokenization for High-Resolution Object Detectioncited by 0