mlsanityData to 14 Oct 2026

Decoder Denoising Pretraining for Semantic Segmentation

arXiv 2205.11423 ↗First posted 23 May 2022cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretraining is commonly used to improve the label-efficiency of segmentation models. Typically, the encoder of a segmentation model is pretrained as a classifier and the decoder is randomly initialized. Here, we argue that random initialization of the decoder can be suboptimal, especially when few labeled examples are available. We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder. We find that decoder denoising pretraining on the ImageNet dataset strongly outperforms encoder-only supervised pretraining. Despite its simplicity, decoder denoising pretraining achieves state-of-the-art results on label-efficient semantic segmentation and offers considerable gains on the Cityscapes, Pascal Context, and ADE20K datasets.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 80 of its 102 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

80 of 102 references.

References · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  11. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2014Going Deeper with Convolutionscited by 4,241
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  13. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  14. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  15. 2017Non-local Neural Networkscited by 2,293
  16. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  17. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  19. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  20. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  21. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  22. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  23. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  24. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  25. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  26. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  27. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  28. 2016Colorful Image Colorizationcited by 1,114
  29. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  30. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  31. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  32. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  33. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  34. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  35. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  36. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  37. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  38. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 8; 2024: 3; 2025: 3.

20 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  2. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  3. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  4. 2023Contrastive Conditional Latent Diffusion for Audio-visual Segmentationcited by 31
  5. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  6. 2022Few-shot Image Generation with Diffusion Modelscited by 21
  7. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  8. 2022DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Modelscited by 12
  9. 2023DDMT: Denoising Diffusion Mask Transformer Models for Multivariate Time Series Anomaly Detectioncited by 10
  10. 2022RangeAugment: Efficient Online Augmentation with Range Learningcited by 8
  11. 2023Unsupervised Discovery of 3D Hierarchical Structure with Generative Diffusion Featurescited by 4
  12. 2023C-DARL: Contrastive diffusion adversarial representation learning for label-free blood vessel segmentationcited by 4
8 more
  1. 2023Time Series Diffusion Method: A Denoising Diffusion Probabilistic Model for Vibration Signal Generationcited by 4
  2. 2025Attention in Diffusion Model: A Surveycited by 4
  3. 2023Pre-Training with Diffusion models for Dental Radiography segmentationcited by 3
  4. 2023A Deep Learning-based Compression and Classification Technique for Whole Slide Histopathology Imagescited by 2
  5. 2024Self-supervised pre-training with diffusion model for few-shot landmark detection in x-ray imagescited by 2
  6. 2025Unified 3D MRI Representations via Sequence-Invariant Contrastive Learningcited by 1
  7. 2025Spatial-Spectral Diffusion Contrastive Representation Network for Hyperspectral Image Classificationcited by 1
  8. 2024Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecturecited by 0