# ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation

arXiv: https://arxiv.org/abs/2108.12382 · HTML: https://mlsanity.com/paper/2108.12382/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)
- [ParseNet: Looking Wider to See Better](https://mlsanity.com/paper/1506.04579/) (2015)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)
- [Disentangled Non-Local Neural Networks](https://mlsanity.com/paper/2006.06668/) (2020)
- [Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing](https://mlsanity.com/paper/1703.05446/) (2017)
- [ACFNet: Attentional Class Feature Network for Semantic Segmentation](https://mlsanity.com/paper/1909.09408/) (2019)
- [Devil in the Details: Towards Accurate Single and Multiple Human Parsing](https://mlsanity.com/paper/1809.05996/) (2018)
- [Semantic Correlation Promoted Shape-Variant Context for Segmentation](https://mlsanity.com/paper/1909.02651/) (2019)
- [Adaptive Context Network for Scene Parsing](https://mlsanity.com/paper/1911.01664/) (2019)

## Built on by (most cited first, 30 of 40)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [SegViT: Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2210.05844/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2303.08600/) (2023)
- [SeMask: Semantically Masked Transformers for Semantic Segmentation](https://mlsanity.com/paper/2112.12782/) (2021)
- [StructToken : Rethinking Semantic Segmentation with Structural Prior](https://mlsanity.com/paper/2203.12612/) (2022)
- [Semantic Diffusion Network for Semantic Segmentation](https://mlsanity.com/paper/2302.02057/) (2023)
- [CDGNet: Class Distribution Guided Network for Human Parsing](https://mlsanity.com/paper/2111.14173/) (2021)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2306.06289/) (2023)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation](https://mlsanity.com/paper/2306.06842/) (2023)
- [Deep Learning Technique for Human Parsing: A Survey and Outlook](https://mlsanity.com/paper/2301.00394/) (2023)
- [Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness](https://mlsanity.com/paper/2503.18445/) (2025)
- [LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning](https://mlsanity.com/paper/2309.13556/) (2023)
- [Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2401.15261/) (2024)
- [Multi-spectral Class Center Network for Face Manipulation Detection and Localization](https://mlsanity.com/paper/2305.10794/) (2023)
- [MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2209.04471/) (2022)
- [MVP-SEG: Multi-View Prompt Learning for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2304.06957/) (2023)
- [MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2503.06700/) (2025)
- [Boosting Night-time Scene Parsing with Learnable Frequency](https://mlsanity.com/paper/2208.14241/) (2022)
- [Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning](https://mlsanity.com/paper/2503.17914/) (2025)
- [Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation](https://mlsanity.com/paper/2309.08020/) (2023)
- [IDRNet: Intervention-Driven Relation Network for Semantic Segmentation](https://mlsanity.com/paper/2310.10755/) (2023)
- [LOGCAN++: Adaptive Local-global class-aware network for semantic segmentation of remote sensing imagery](https://mlsanity.com/paper/2406.16502/) (2024)
- [Class-level Multiple Distributions Representation are Necessary for Semantic Segmentation](https://mlsanity.com/paper/2303.08029/) (2023)
- [PRSeg: A Lightweight Patch Rotate MLP Decoder for Semantic Segmentation](https://mlsanity.com/paper/2305.00671/) (2023)
- [SSSegmenation: An Open Source Supervised Semantic Segmentation Toolbox Based on PyTorch](https://mlsanity.com/paper/2305.17091/) (2023)
- [SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation](https://mlsanity.com/paper/2405.06525/) (2024)

## Abstract (arXiv)

Co-occurrent visual pattern makes aggregating contextual information a common paradigm to enhance the pixel representation for semantic image segmentation. The existing approaches focus on modeling the context from the perspective of the whole image, i.e., aggregating the image-level contextual information. Despite impressive, these methods weaken the significance of the pixel representations of the same category, i.e., the semantic-level contextual information. To address this, this paper proposes to augment the pixel representations by aggregating the image-level and semantic-level contextual information, respectively. First, an image-level context module is designed to capture the contextual information for each pixel in the whole image. Second, we aggregate the representations of the same category for each pixel where the category regions are learned under the supervision of the ground-truth segmentation. Third, we compute the similarities between each pixel representation and the image-level contextual information, the semantic-level contextual information, respectively. At last, a pixel representation is augmented by weighted aggregating both the image-level contextual information and the semantic-level contextual information with the similarities as the weights. Integrating the image-level and semantic-level context allows this paper to report state-of-the-art accuracy on four benchmarks, i.e., ADE20K, LIP, COCOStuff and Cityscapes.