mlsanityData to 7 Oct 2026

Diffusion Models as Masked Autoencoders

arXiv 2304.03283 ↗First posted 6 Apr 2023cs.CVcited by 71

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

There has been a longstanding belief that generation can facilitate a true understanding of visual data. In line with this, we revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE). Our approach is capable of (i) serving as a strong initialization for downstream recognition tasks, (ii) conducting high-quality image inpainting, and (iii) being effortlessly extended to video where it produces state-of-the-art classification accuracy. We further perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 85 of its 99 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

85 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2016Layer Normalizationcited by 5,579
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Neural Discrete Representation Learningcited by 4,650
  13. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  14. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  15. 2013Auto-Encoding Variational Bayescited by 3,980
  16. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  17. 2022A ConvNet for the 2020scited by 3,545
  18. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  19. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  20. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  21. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  22. 2022Video Diffusion Modelscited by 2,280
  23. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  24. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  25. 2018SlowFast Networks for Video Recognitioncited by 1,862
  26. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  27. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  28. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  29. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  30. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  31. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  32. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  33. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  34. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  35. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  36. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  37. 2016Colorful Image Colorizationcited by 1,113
  38. 2018Self-Attention with Relative Position Representationscited by 1,112

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 24; 2025: 23; 2026: 14.

71 papers.

Sort:

By area:

  1. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  2. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  3. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  4. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  5. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  6. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  7. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  8. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  9. 2023Point Cloud Pre-training with Diffusion Modelscited by 47
  10. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  11. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  12. 2024Masked Image Modeling: A Surveycited by 28
38 more
  1. 2024Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraintscited by 26
  2. 2023Hierarchical Masked 3D Diffusion Model for Video Outpaintingcited by 25
  3. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  4. 2024Is Tokenization Needed for Masked Particle Modelling?cited by 20
  5. 2024Lazy Diffusion Transformer for Interactive Image Editingcited by 19
  6. 2025GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancerscited by 16
  7. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learnerscited by 14
  8. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  9. 2023MonoDiffusion: Self-Supervised Monocular Depth Estimation Using Diffusion Modelcited by 12
  10. 2024Harnessing Massive Satellite Imagery with Efficient Masked Image Modelingcited by 12
  11. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  12. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  13. 2025How far can we go with ImageNet for Text-to-Image generation?cited by 10
  14. 2023Diffusion-based generation of Histopathological Whole Slide Images at a Gigapixel scalecited by 9
  15. 2024Diffusion Models For Multi-Modal Generative Modelingcited by 9
  16. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generationcited by 8
  17. 2025AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Modelscited by 8
  18. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  19. 20254D Visual Pre-training for Robot Learningcited by 8
  20. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  21. 2024LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Imagecited by 7
  22. 2024MAEDiff: Masked Autoencoder-enhanced Diffusion Models for Unsupervised Anomaly Detection in Brain Imagescited by 6
  23. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  24. 2023Reconstructing Historical Climate Fields With Deep Learningcited by 4
  25. 2024Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priorscited by 4
  26. 2024ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipationcited by 4
  27. 2025Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Modelscited by 4
  28. 2024Physical Consistency Bridges Heterogeneous Data in Molecular Multi-Task Learningcited by 3
  29. 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learningcited by 3
  30. 2025Focus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classificationcited by 3
  31. 2023Diffusion Models as Masked Audio-Video Learnerscited by 2
  32. 2024Unified Auto-Encoding with Masked Diffusioncited by 2
  33. 2025Masked Clustering Prediction for Unsupervised Point Cloud Pre-trainingcited by 2
  34. 2026Trajectory Forcing: Structure-First Generation with Controllable Semantic Trajectoriescited by 2
  35. 2024Predicting Long-horizon Futures by Conditioning on Geometry and Timecited by 1
  36. 2025Graffe: Graph Representation Learning via Diffusion Probabilistic Modelscited by 1
  37. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  38. 2025Diffusion Autoencoders with Perceivers for Long, Irregular and Multimodal Astronomical Sequencescited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Masked Diffusion Models Are Fast Distribution Learners
  2. 2025Latent Diffusion Models with Masked AutoEncoders
  3. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learners
  4. 2024Simplified and Generalized Masked Diffusion for Discrete Data
  5. 2022Representation Learning with Diffusion Models
  6. 2025Diffusion Transformers with Representation Autoencoders
  7. 2024Diffusion Models in Low-Level Vision: A Survey
  8. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representations