# Specialize and Fuse: Pyramidal Output Representation for Semantic Segmentation

arXiv: https://arxiv.org/abs/2108.01866 · HTML: https://mlsanity.com/paper/2108.01866/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)
- [Expectation-Maximization Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1907.13426/) (2019)
- [Disentangled Non-Local Neural Networks](https://mlsanity.com/paper/2006.06668/) (2020)
- [Context Prior for Scene Segmentation](https://mlsanity.com/paper/2004.01547/) (2020)
- [Strip Pooling: Rethinking Spatial Pooling for Scene Parsing](https://mlsanity.com/paper/2003.13328/) (2020)
- [Boundary-Aware Feature Propagation for Scene Segmentation](https://mlsanity.com/paper/1909.00179/) (2019)
- [Improving Semantic Segmentation via Decoupled Body and Edge Supervision](https://mlsanity.com/paper/2007.10035/) (2020)
- [ACFNet: Attentional Class Feature Network for Semantic Segmentation](https://mlsanity.com/paper/1909.09408/) (2019)
- [Semantic Correlation Promoted Shape-Variant Context for Segmentation](https://mlsanity.com/paper/1909.02651/) (2019)
- [Dual Graph Convolutional Network for Semantic Segmentation](https://mlsanity.com/paper/1909.06121/) (2019)
- [GFF: Gated Fully Fusion for Semantic Segmentation](https://mlsanity.com/paper/1904.01803/) (2019)
- [Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade](https://mlsanity.com/paper/1704.01344/) (2017)
- [Adaptive Context Network for Scene Parsing](https://mlsanity.com/paper/1911.01664/) (2019)
- [Class-wise Dynamic Graph Convolution for Semantic Segmentation](https://mlsanity.com/paper/2007.09690/) (2020)
- [Learning to Predict Context-adaptive Convolution for Semantic Segmentation](https://mlsanity.com/paper/2004.08222/) (2020)
- [Quadtree Generating Networks: Efficient Hierarchical Scene Parsing with Sparse Convolutions](https://mlsanity.com/paper/1907.11821/) (2019)

## Built on by (most cited first, 7 of 7)

- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation](https://mlsanity.com/paper/2306.06842/) (2023)
- [LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning](https://mlsanity.com/paper/2309.13556/) (2023)
- [Data Efficient 3D Learner via Knowledge Transferred from 2D Model](https://mlsanity.com/paper/2203.08479/) (2022)

## Abstract (arXiv)

We present a novel pyramidal output representation to ensure parsimony with our "specialize and fuse" process for semantic segmentation. A pyramidal "output" representation consists of coarse-to-fine levels, where each level is "specialize" in a different class distribution (e.g., more stuff than things classes at coarser levels). Two types of pyramidal outputs (i.e., unity and semantic pyramid) are "fused" into the final semantic output, where the unity pyramid indicates unity-cells (i.e., all pixels in such cell share the same semantic label). The process ensures parsimony by predicting a relatively small number of labels for unity-cells (e.g., a large cell of grass) to build the final semantic output. In addition to the "output" representation, we design a coarse-to-fine contextual module to aggregate the "features" representation from different levels. We validate the effectiveness of each key module in our method through comprehensive ablation studies. Finally, our approach achieves state-of-the-art performance on three widely-used semantic segmentation datasets -- ADE20K, COCO-Stuff, and Pascal-Context.