PAUMER: Patch Pausing Transformer for Semantic Segmentation
Not on the map. Why?
Abstract (arXiv)
We study the problem of improving the efficiency of segmentation transformers by using disparate amounts of computation for different parts of the image. Our method, PAUMER, accomplishes this by pausing computation for patches that are deemed to not need any more computation before the final decoder. We use the entropy of predictions computed from intermediate activations as the pausing criterion, and find this aligns well with semantics of the image. Our method has a unique advantage that a single network trained with the proposed strategy can be effortlessly adapted at inference to various run-time requirements by modulating its pausing parameters. On two standard segmentation datasets, Cityscapes and ADE20K, we show that our method operates with about a 50% higher throughput with an mIoU drop of about 0.65% and 4.6% respectively.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 1 Nov 2023.
Builds on
31 of 36 references.
References · 31
- 2017Attention Is All You Needcited by 59,932
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
- 2021Transformers in Vision: A Surveycited by 1,009
- 2021Segmenter: Transformer for Semantic Segmentationcited by 855
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
19 more
- 2020Efficient Transformers: A Surveycited by 784
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
- 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2021Dynamic Neural Networks: A Surveycited by 405
- 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
- 2018Slimmable Neural Networkscited by 301
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
- 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
- 2021Co-Scale Conv-Attentional Image Transformerscited by 224
- 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
- 2016Spatially Adaptive Computation Time for Residual Networkscited by 118
- 2015Fast Algorithms for Convolutional Neural Networkscited by 102
- 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
- 2021Token Pooling in Vision Transformerscited by 68
- 2017Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascadecited by 53
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.