Sparse Refinement for Efficient High-Resolution Semantic Segmentation
Not on the map. Why?
Abstract (arXiv)
Semantic segmentation empowers numerous real-world applications, such as autonomous driving and augmented/mixed reality. These applications often operate on high-resolution images (e.g., 8 megapixels) to capture the fine details. However, this comes at the cost of considerable computational complexity, hindering the deployment in latency-sensitive scenarios. In this paper, we introduce SparseRefine, a novel approach that enhances dense low-resolution predictions with sparse high-resolution refinements. Based on coarse low-resolution outputs, SparseRefine first uses an entropy selector to identify a sparse set of pixels with high entropy. It then employs a sparse feature extractor to efficiently generate the refinements for those pixels of interest. Finally, it leverages a gated ensembler to apply these sparse refinements to the initial coarse predictions. SparseRefine can be seamlessly integrated into any existing semantic segmentation model, regardless of CNN- or ViT-based. SparseRefine achieves significant speedup: 1.5 to 3.7 times when applied to HRNet-W48, SegFormer-B5, Mask2Former-T/L and SegNeXt-L on Cityscapes, with negligible to no loss of accuracy. Our "dense+sparse" paradigm paves the way for efficient high-resolution visual computing.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 87 of its 108 references that are in the map. First posted on arXiv 26 Jul 2024.
Builds on
87 of 108 references.
References · 87
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
38 more
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
- 2016Pyramid Scene Parsing Networkcited by 2,472
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
- 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
- 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
- 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
- 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
- 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
- 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2018Dual Attention Network for Scene Segmentationcited by 1,001
- 2022Token Merging: Your ViT But Fastercited by 995
- 2021Multiscale Vision Transformerscited by 896
- 2021Segmenter: Transformer for Semantic Segmentationcited by 855
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
- 2018PointCNN: Convolution On X-Transformed Pointscited by 764
- 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
- 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
- 2019Panoptic Feature Pyramid Networkscited by 644
- 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
- 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
- 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
- 2019PointRend: Image Segmentation as Renderingcited by 439
- 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
- 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
- 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
- 2020BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentationcited by 294
- 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
50 most cited of 87 All 87 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.