# Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation

arXiv: https://arxiv.org/abs/1903.02120 · HTML: https://mlsanity.com/paper/1903.02120/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2019.

## Built on by (most cited first, 30 of 34)

- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)
- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [SAPA: Similarity-Aware Point Affiliation for Feature Upsampling](https://mlsanity.com/paper/2209.12866/) (2022)
- [HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection](https://mlsanity.com/paper/2301.03036/) (2023)
- [More than Encoder: Introducing Transformer Decoder to Upsample](https://mlsanity.com/paper/2106.10637/) (2021)
- [FADE: Fusing the Assets of Decoder and Encoder for Task-Agnostic Upsampling](https://mlsanity.com/paper/2207.10392/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms](https://mlsanity.com/paper/2103.02433/) (2021)
- [Instance and Panoptic Segmentation Using Conditional Convolutions](https://mlsanity.com/paper/2102.03026/) (2021)
- [Graph Attention Layer Evolves Semantic Segmentation for Road Pothole Detection: A Benchmark and Algorithms](https://mlsanity.com/paper/2109.02711/) (2021)
- [MaskRange: A Mask-classification Model for Range-view based LiDAR Segmentation](https://mlsanity.com/paper/2206.12073/) (2022)
- [Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2107.14428/) (2021)
- [Attention guided global enhancement and local refinement network for semantic segmentation](https://mlsanity.com/paper/2204.04363/) (2022)
- [Graph Reasoning Transformer for Image Parsing](https://mlsanity.com/paper/2209.09545/) (2022)
- [ORDNet: Capturing Omni-Range Dependencies for Scene Parsing](https://mlsanity.com/paper/2101.03929/) (2021)
- [Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation](https://mlsanity.com/paper/2110.01939/) (2021)
- [PLGAN: Generative Adversarial Networks for Power-Line Segmentation in Aerial Images](https://mlsanity.com/paper/2204.07243/) (2022)
- [A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling](https://mlsanity.com/paper/2407.02283/) (2024)
- [Multi-task Fusion for Efficient Panoptic-Part Segmentation](https://mlsanity.com/paper/2212.07671/) (2022)
- [Dual-Flattening Transformers through Decomposed Row and Column Queries for Semantic Segmentation](https://mlsanity.com/paper/2201.09139/) (2022)
- [Category Feature Transformer for Semantic Segmentation](https://mlsanity.com/paper/2308.05581/) (2023)
- [FADE: A Task-Agnostic Upsampling Operator for Encoder-Decoder Architectures](https://mlsanity.com/paper/2407.13500/) (2024)
- [Spatial Frequency Modulation for Semantic Segmentation](https://mlsanity.com/paper/2507.11893/) (2025)
- [LDRNet: Enabling Real-time Document Localization on Mobile Devices](https://mlsanity.com/paper/2206.02136/) (2022)
- [JPPF: Multi-task Fusion for Consistent Panoptic-Part Segmentation](https://mlsanity.com/paper/2311.18618/) (2023)
- [BACS: Background Aware Continual Semantic Segmentation](https://mlsanity.com/paper/2404.13148/) (2024)
- [A Novel Upsampling and Context Convolution for Image Semantic Segmentation](https://mlsanity.com/paper/2103.11110/) (2021)
- [A De-raining semantic segmentation network for real-time foreground segmentation](https://mlsanity.com/paper/2104.07877/) (2021)
- [Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task](https://mlsanity.com/paper/2105.13531/) (2021)

## Abstract (arXiv)

Recent semantic segmentation methods exploit encoder-decoder architectures to produce the desired pixel-wise segmentation prediction. The last layer of the decoders is typically a bilinear upsampling procedure to recover the final pixel-wise prediction. We empirically show that this oversimple and data-independent bilinear upsampling may lead to sub-optimal results. In this work, we propose a data-dependent upsampling (DUpsampling) to replace bilinear, which takes advantages of the redundancy in the label space of semantic segmentation and is able to recover the pixel-wise prediction from low-resolution outputs of CNNs. The main advantage of the new upsampling layer lies in that with a relatively lower-resolution feature map such as frac116 or frac132 of the input size, we can achieve even better segmentation accuracy, significantly reducing computation complexity. This is made possible by 1) the new upsampling layer's much improved reconstruction capability; and more importantly 2) the DUpsampling based decoder's flexibility in leveraging almost arbitrary combinations of the CNN encoders' features. Experiments demonstrate that our proposed decoder outperforms the state-of-the-art decoder, with only ~20% of computation. Finally, without any post-processing, the framework equipped with our proposed decoder achieves new state-of-the-art performance on two datasets: 88.1% mIOU on PASCAL VOC with 30% computation of the previously best model; and 52.5% mIOU on PASCAL Context.