mlsanityData to 14 Oct 2026

Matryoshka Diffusion Models

arXiv 2310.15111 ↗First posted 23 Oct 2023cs.CV cs.LGcited by 72

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models are the de facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges. Existing methods often resort to training cascaded models in pixel space or using a downsampled latent space of a separately trained auto-encoder. In this paper, we introduce Matryoshka Diffusion Models(MDM), an end-to-end framework for high-resolution image and video synthesis. We propose a diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small-scale inputs are nested within those of large scales. In addition, MDM enables a progressive training schedule from lower to higher resolutions, which leads to significant improvements in optimization for high-resolution generation. We demonstrate the effectiveness of our approach on various benchmarks, including class-conditioned image generation, high-resolution text-to-image, and text-to-video applications. Remarkably, we can train a single pixel-space model at resolutions of up to 1024x1024 pixels, demonstrating strong zero-shot generalization using the CC12M dataset, which contains only 12 million images. Our code is released at https://github.com/apple/ml-mdm

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 49 of its 56 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

49 of 56 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  2. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  3. 2023On the Importance of Noise Scheduling for Diffusion Models26 Jan
  4. 2022Scalable Diffusion Models with Transformers19 Dec
  5. 2022f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation10 Oct
  6. 2022Pyramidal Denoising Diffusion Probabilistic Models3 Aug
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  9. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  10. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
25 more
  1. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  10. 2016Pixel Recurrent Neural Networkscited by 816
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  13. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  16. 2022Matryoshka Representation Learningcited by 282
  17. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  18. 2022Scalable Adaptive Computation for Iterative Generationcited by 195
  19. 2019Scaling Autoregressive Video Modelscited by 190
  20. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 180
  21. 2022Generative Modelling With Inverse Heat Dissipationcited by 167
  22. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  23. 2016Video Pixel Networkscited by 108
  24. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 58
  25. 2023Learning multi-scale local conditional probability models of imagescited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 32; 2025: 25; 2026: 10.

72 papers.

Sort:

By area:

  1. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  2. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  3. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  4. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  5. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  6. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
  7. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 97
  8. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 75
  9. 2024Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptationcited by 51
  10. 2024Steering Rectified Flow Models in the Vector Field for Controlled Image Generationcited by 39
  11. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 38
  12. 2024Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalizationcited by 36
38 more
  1. 2024RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Modelscited by 36
  2. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenizationcited by 36
  3. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generationcited by 32
  4. 2025FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generationcited by 32
  5. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  6. 2025Latent Space Super-Resolution for Higher-Resolution Image Generation with Diffusion Modelscited by 25
  7. 2024Hierarchical Patch Diffusion Models for High-Resolution Video Generationcited by 24
  8. 2024Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellencycited by 22
  9. 2024Upsample Guidance: Scale Up Diffusion Models without Trainingcited by 20
  10. 2024Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problemscited by 20
  11. 2023Synthetically Enhanced: Unveiling Synthetic Data's Potential in Medical Imaging Researchcited by 18
  12. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  13. 2024Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Modelscited by 17
  14. 2025Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategycited by 16
  15. 2024Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanismscited by 15
  16. 2024Spectral Image Tokenizercited by 15
  17. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generationcited by 14
  18. 2024Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Modelscited by 14
  19. 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
  20. 2024Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modelingcited by 13
  21. 2025Scale-wise Distillation of Diffusion Modelscited by 11
  22. 2025How far can we go with ImageNet for Text-to-Image generation?cited by 10
  23. 2024∞-Brush: Controllable Large Image Synthesis with Diffusion Models in Infinite Dimensionscited by 8
  24. 2024Preference Adaptive and Sequential Text-to-Image Generationcited by 8
  25. 2026Asymmetric Flow Modelscited by 8
  26. 2024MatMamba: A Matryoshka State Space Modelcited by 7
  27. 2025Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesiscited by 7
  28. 2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 7
  29. 2025ImageDoctor: Diagnosing Text-to-Image Generation via Grounded Image Reasoningcited by 7
  30. 2024E(3)-equivariant models cannot learn chirality: Field-based molecular generationcited by 6
  31. 2025Improving Progressive Generation with Decomposable Flow Matchingcited by 6
  32. 2024AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generationcited by 5
  33. 2025ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuningcited by 5
  34. 2025NAMI: Efficient Image Generation via Bridged Progressive Rectified Flow Transformerscited by 4
  35. 2025Pyramidal Patchification Flow for Visual Generationcited by 4
  36. 2025Elastic ViTs from Pretrained Models without Retrainingcited by 4
  37. 2026DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformerscited by 4
  38. 2026Scale Space Diffusioncited by 4

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance
  2. 2024MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning
  3. 2022Representation Learning with Diffusion Models
  4. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
  5. 2026Efficient and Training-Free Single-Image Diffusion Models
  6. 2023Diffusion Models Without Attention
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  8. 2022On Distillation of Guided Diffusion Models