# Hierarchical Multi-Scale Attention for Semantic Segmentation

arXiv: https://arxiv.org/abs/2005.10821 · HTML: https://mlsanity.com/paper/2005.10821/

## Key facts

- Cited by 183 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 30 of 183)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting](https://mlsanity.com/paper/2403.12722/) (2024)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)
- [Hyperbolic Image Segmentation](https://mlsanity.com/paper/2203.05898/) (2022)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [Deep Dense Multi-scale Network for Snow Removal Using Semantic and Geometric Priors](https://mlsanity.com/paper/2103.11298/) (2021)
- [ESS: Learning Event-based Semantic Segmentation from Still Images](https://mlsanity.com/paper/2203.10016/) (2022)
- [WildNet: Learning Domain Generalized Semantic Segmentation from the Wild](https://mlsanity.com/paper/2204.01446/) (2022)
- [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Learning Non-target Knowledge for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2205.04903/) (2022)
- [SeMask: Semantically Masked Transformers for Semantic Segmentation](https://mlsanity.com/paper/2112.12782/) (2021)
- [Towards Label-free Scene Understanding by Vision Foundation Models](https://mlsanity.com/paper/2306.03899/) (2023)
- [Global Features are All You Need for Image Retrieval and Reranking](https://mlsanity.com/paper/2308.06954/) (2023)
- [Polarized Self-Attention: Towards High-quality Pixel-wise Regression](https://mlsanity.com/paper/2107.00782/) (2021)
- [EDTER: Edge Detection with Transformer](https://mlsanity.com/paper/2203.08566/) (2022)
- [PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation](https://mlsanity.com/paper/2101.06175/) (2021)
- [PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain](https://mlsanity.com/paper/2306.04557/) (2023)
- [LOHO: Latent Optimization of Hairstyles via Orthogonalization](https://mlsanity.com/paper/2103.03891/) (2021)
- [MGNet: Monocular Geometric Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2206.13199/) (2022)
- [Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering](https://mlsanity.com/paper/2408.09702/) (2024)

## Abstract (arXiv)

Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).