mlsanityData to 7 Oct 2026

Scalable Vision Transformers with Hierarchical Pooling

arXiv 2103.10619 ↗First posted 19 Mar 2021cs.CVcited by 84

Not on the map. Why?

Abstract (arXiv)

The recently proposed Visual image Transformers (ViT) with pure attention have achieved promising performance on image recognition tasks, such as image classification. However, the routine of the current ViT model is to maintain a full-length patch sequence during inference, which is redundant and lacks hierarchical representation. To this end, we propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length and hence reduces the computational cost, analogous to the feature maps downsampling in Convolutional Neural Networks (CNNs). It brings a great benefit that we can increase the model capacity by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity due to the reduced sequence length. Moreover, we empirically find that the average pooled visual tokens contain more discriminative information than the single class token. To demonstrate the improved scalability of our HVT, we conduct extensive experiments on the image classification task. With comparable FLOPs, our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets. Code is available at https://github.com/MonashAI/HVT

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

44 of 53 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2016Layer Normalizationcited by 5,579
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
32 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  3. 2016Wide Residual Networkscited by 3,057
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  13. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  14. 2019Are Sixteen Heads Really Better than One?cited by 803
  15. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  18. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  19. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  20. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  21. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  22. 2019Axial Attention in Multidimensional Transformerscited by 350
  23. 2021Random Feature Attentioncited by 350
  24. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  25. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  26. 2019Local Relation Networks for Image Recognitioncited by 210
  27. 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184
  28. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  29. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  30. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  31. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  32. 2019Fully Quantized Transformer for Machine Translationcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 23; 2023: 24; 2024: 17; 2025: 8; 2026: 5.

84 papers.

Sort:

By area:

  1. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  2. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  4. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  5. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  6. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  7. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  8. 2022Unified Visual Transformer Compressioncited by 79
  9. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  10. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  11. 2021Gaze Estimation using Transformercited by 68
  12. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
38 more
  1. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  2. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  3. 2022Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Versioncited by 59
  4. 2024A Survey on Transformer Compressioncited by 44
  5. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  6. 2022Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learningcited by 27
  7. 2023Stitchable Neural Networkscited by 26
  8. 2023Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentationcited by 24
  9. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 23
  10. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  11. 2023MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentationcited by 22
  12. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  13. 2022Revealing Occlusions with 4D Neural Fieldscited by 17
  14. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  15. 2022Dynamic Focus-aware Positional Queries for Semantic Segmentationcited by 16
  16. 2022An Efficient Spatio-Temporal Pyramid Transformer for Action Detectioncited by 16
  17. 2023Hierarchical Visual Primitive Experts for Compositional Zero-Shot Learningcited by 16
  18. 2022TransMatting: Enhancing Transparent Objects Matting with Transformerscited by 15
  19. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  20. 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
  21. 2023Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attentioncited by 12
  22. 2023Explainability of Vision Transformers: A Comprehensive Review and New Perspectivescited by 12
  23. 2023ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretrainingcited by 11
  24. 2022Multi-Tailed Vision Transformer for Efficient Inferencecited by 10
  25. 2022Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Datacited by 10
  26. 2024UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformercited by 9
  27. 2024PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inferencecited by 9
  28. 2023HM-Conformer: A Conformer-based audio deepfake detection system with hierarchical pooling and multi-level classification token aggregation methodscited by 8
  29. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  30. 2022Data Level Lottery Ticket Hypothesis for Vision Transformerscited by 7
  31. 2022Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Tokencited by 7
  32. 2023Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiencycited by 7
  33. 2024You Only Need Less Attention at Each Stage in Vision Transformerscited by 7
  34. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  35. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
  36. 2022FocusFormer: Focusing on What We Need via Architecture Samplercited by 5
  37. 2021End-to-end One-shot Human Parsingcited by 4
  38. 2021Sparse Fusion for Multimodal Transformerscited by 4

50 most cited of 84 All 84 (JSON)