# The Devil is in the Decoder: Classification, Regression and GANs

arXiv: https://arxiv.org/abs/1707.05847 · HTML: https://mlsanity.com/paper/1707.05847/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2017.

## Built on by (most cited first, 22 of 22)

- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Unpaired Deep Image Deraining Using Dual Contrastive Learning](https://mlsanity.com/paper/2109.02973/) (2021)
- [Towards Unified Deep Image Deraining: A Survey and A New Benchmark](https://mlsanity.com/paper/2310.03535/) (2023)
- [SAPA: Similarity-Aware Point Affiliation for Feature Upsampling](https://mlsanity.com/paper/2209.12866/) (2022)
- [Prototype Guided Federated Learning of Visual Feature Representations](https://mlsanity.com/paper/2105.08982/) (2021)
- [Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances](https://mlsanity.com/paper/2209.13131/) (2022)
- [Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses](https://mlsanity.com/paper/2104.08717/) (2021)
- [HS3-Bench: A Benchmark and Strong Baseline for Hyperspectral Semantic Segmentation in Driving Scenarios](https://mlsanity.com/paper/2409.11205/) (2024)
- [Channel Consistency Prior and Self-Reconstruction Strategy Based Unsupervised Image Deraining](https://mlsanity.com/paper/2503.18703/) (2025)
- [Double Encoder-Decoder Networks for Gastrointestinal Polyp Segmentation](https://mlsanity.com/paper/2110.01939/) (2021)
- [Context-Aware Scene Prediction Network (CASPNet)](https://mlsanity.com/paper/2201.06933/) (2022)
- [A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling](https://mlsanity.com/paper/2407.02283/) (2024)
- [MRI Scan Synthesis Methods based on Clustering and Pix2Pix](https://mlsanity.com/paper/2312.05176/) (2023)
- [Improved-Mask R-CNN: Towards an Accurate Generic MSK MRI instance segmentation platform (Data from the Osteoarthritis Initiative)](https://mlsanity.com/paper/2107.12889/) (2021)
- [Monocular Depth Estimation Primed by Salient Point Detection and Normalized Hessian Loss](https://mlsanity.com/paper/2108.11098/) (2021)
- [DXM-TransFuse U-net: Dual Cross-Modal Transformer Fusion U-net for Automated Nerve Identification](https://mlsanity.com/paper/2202.13304/) (2022)
- [VoloGAN: Adversarial Domain Adaptation for Synthetic Depth Data](https://mlsanity.com/paper/2207.09204/) (2022)
- [On Point Affiliation in Feature Upsampling](https://mlsanity.com/paper/2307.08198/) (2023)
- [HCFormer: Unified Image Segmentation with Hierarchical Clustering](https://mlsanity.com/paper/2205.09949/) (2022)
- [Deformably-Scaled Transposed Convolution](https://mlsanity.com/paper/2210.09446/) (2022)
- [Learning to Detect Touches on Cluttered Tables](https://mlsanity.com/paper/2304.04687/) (2023)
- [Feature Aggregation in Joint Sound Classification and Localization Neural Networks](https://mlsanity.com/paper/2310.19063/) (2023)

## Abstract (arXiv)

Many machine vision applications, such as semantic segmentation and depth prediction, require predictions for every pixel of the input image. Models for such problems usually consist of encoders which decrease spatial resolution while learning a high-dimensional representation, followed by decoders who recover the original input resolution and result in low-dimensional predictions. While encoders have been studied rigorously, relatively few studies address the decoder side. This paper presents an extensive comparison of a variety of decoders for a variety of pixel-wise tasks ranging from classification, regression to synthesis. Our contributions are: (1) Decoders matter: we observe significant variance in results between different types of decoders on various problems. (2) We introduce new residual-like connections for decoders. (3) We introduce a novel decoder: bilinear additive upsampling. (4) We explore prediction artifacts.