# Attention to Scale: Scale-aware Semantic Image Segmentation

arXiv: https://arxiv.org/abs/1511.03339 · HTML: https://mlsanity.com/paper/1511.03339/

## Key facts

- Cited by 169 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2015.

## Built on by (most cited first, 30 of 169)

- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2111.12903/) (2021)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Self-Support Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2207.11549/) (2022)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation](https://mlsanity.com/paper/2107.11279/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS](https://mlsanity.com/paper/2101.07172/) (2021)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [PLIP: Language-Image Pre-training for Person Representation Learning](https://mlsanity.com/paper/2305.08386/) (2023)
- [Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2108.11819/) (2021)
- [A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation](https://mlsanity.com/paper/2107.12435/) (2021)
- [ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation](https://mlsanity.com/paper/2108.12382/) (2021)
- [An efficient encoder-decoder architecture with top-down attention for speech separation](https://mlsanity.com/paper/2209.15200/) (2022)
- [Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2203.14098/) (2022)
- [FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation](https://mlsanity.com/paper/2103.17235/) (2021)
- [Structural and Statistical Texture Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2305.03944/) (2023)
- [PaddleSeg: A High-Efficient Development Toolkit for Image Segmentation](https://mlsanity.com/paper/2101.06175/) (2021)
- [Progressive Semantic Segmentation](https://mlsanity.com/paper/2104.03778/) (2021)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [Geometric Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16694/) (2021)
- [Unbiased Subclass Regularization for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.10026/) (2022)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [OV-PARTS: Towards Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2310.05107/) (2023)
- [Robust Attentive Deep Neural Network for Exposing GAN-generated Faces](https://mlsanity.com/paper/2109.02167/) (2021)

## Abstract (arXiv)

Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.