# Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation

arXiv: https://arxiv.org/abs/1909.11065 · HTML: https://mlsanity.com/paper/1909.11065/

## Key facts

- Cited by 576 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2019.

## Built on by (most cited first, 30 of 576)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [HRFormer: High-Resolution Transformer for Dense Prediction](https://mlsanity.com/paper/2110.09408/) (2021)
- [DDP: Diffusion Model for Dense Visual Prediction](https://mlsanity.com/paper/2303.17559/) (2023)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)

## Abstract (arXiv)

In this paper, we address the semantic segmentation problem with a focus on the context aggregation strategy. Our motivation is that the label of a pixel is the category of the object that the pixel belongs to. We present a simple yet effective approach, object-contextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, we learn object regions under the supervision of ground-truth segmentation. Second, we compute the object region representation by aggregating the representations of the pixels lying in the object region. Last, % the representation similarity we compute the relation between each pixel and each object region and augment the representation of each pixel with the object-contextual representation which is a weighted aggregation of all the object region representations according to their relations with the pixel. We empirically demonstrate that the proposed approach achieves competitive performance on various challenging semantic segmentation benchmarks: Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Our submission "HRNet + OCR + SegFix" achieves 1-st place on the Cityscapes leaderboard by the time of submission. Code is available at: https://git.io/openseg and https://git.io/HRNet.OCR. We rephrase the object-contextual representation scheme using the Transformer encoder-decoder framework. The details are presented in~Section3.3.