# Decoder Denoising Pretraining for Semantic Segmentation

arXiv: https://arxiv.org/abs/2205.11423 · HTML: https://mlsanity.com/paper/2205.11423/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 102 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)

## Built on by (most cited first, 20 of 20)

- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [A Generalist Framework for Panoptic Segmentation of Images and Videos](https://mlsanity.com/paper/2210.06366/) (2022)
- [Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models](https://mlsanity.com/paper/2308.16777/) (2023)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [A simple, efficient and scalable contrastive masked autoencoder for learning visual representations](https://mlsanity.com/paper/2210.16870/) (2022)
- [Few-shot Image Generation with Diffusion Models](https://mlsanity.com/paper/2211.03264/) (2022)
- [A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting](https://mlsanity.com/paper/2401.10227/) (2024)
- [DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.08861/) (2022)
- [DDMT: Denoising Diffusion Mask Transformer Models for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2310.08800/) (2023)
- [RangeAugment: Efficient Online Augmentation with Range Learning](https://mlsanity.com/paper/2212.10553/) (2022)
- [Unsupervised Discovery of 3D Hierarchical Structure with Generative Diffusion Features](https://mlsanity.com/paper/2305.00067/) (2023)
- [C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentation](https://mlsanity.com/paper/2308.00193/) (2023)
- [Time Series Diffusion Method: A Denoising Diffusion Probabilistic Model for Vibration Signal Generation](https://mlsanity.com/paper/2312.07981/) (2023)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [Pre-Training with Diffusion models for Dental Radiography segmentation](https://mlsanity.com/paper/2307.14066/) (2023)
- [A Deep Learning-based Compression and Classification Technique for Whole Slide Histopathology Images](https://mlsanity.com/paper/2305.07161/) (2023)
- [Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray images](https://mlsanity.com/paper/2407.18125/) (2024)
- [Unified 3D MRI Representations via Sequence-Invariant Contrastive Learning](https://mlsanity.com/paper/2501.12057/) (2025)
- [Spatial-Spectral Diffusion Contrastive Representation Network for Hyperspectral Image Classification](https://mlsanity.com/paper/2502.19699/) (2025)
- [Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecture](https://mlsanity.com/paper/2407.10733/) (2024)

## Abstract (arXiv)

Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretraining is commonly used to improve the label-efficiency of segmentation models. Typically, the encoder of a segmentation model is pretrained as a classifier and the decoder is randomly initialized. Here, we argue that random initialization of the decoder can be suboptimal, especially when few labeled examples are available. We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder. We find that decoder denoising pretraining on the ImageNet dataset strongly outperforms encoder-only supervised pretraining. Despite its simplicity, decoder denoising pretraining achieves state-of-the-art results on label-efficient semantic segmentation and offers considerable gains on the Cityscapes, Pascal Context, and ADE20K datasets.