# Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion

arXiv: https://arxiv.org/abs/2410.19324 · HTML: https://mlsanity.com/paper/2410.19324/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 47 of its 53 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 47)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion](https://mlsanity.com/paper/2310.02279/) (2023)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)

## Built on by (most cited first, 30 of 75)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [History-Guided Video Diffusion](https://mlsanity.com/paper/2502.06764/) (2025)
- [Generative Modeling via Drifting](https://mlsanity.com/paper/2602.04770/) (2026)
- [Improved Mean Flows: On the Challenges of Fastforward Generative Models](https://mlsanity.com/paper/2512.02012/) (2025)
- [PixelFlow: Pixel-Space Generative Models with Flow](https://mlsanity.com/paper/2504.07963/) (2025)
- [PixelDiT: Pixel Diffusion Transformers for Image Generation](https://mlsanity.com/paper/2511.20645/) (2025)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [One-step Latent-free Image Generation with Pixel Mean Flows](https://mlsanity.com/paper/2601.22158/) (2026)
- [DiP: Taming Diffusion Models in Pixel Space](https://mlsanity.com/paper/2511.18822/) (2025)
- [Fractal Generative Models](https://mlsanity.com/paper/2502.17437/) (2025)
- [Calibrated Multi-Preference Optimization for Aligning Diffusion Models](https://mlsanity.com/paper/2502.02588/) (2025)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation](https://mlsanity.com/paper/2602.11401/) (2026)
- [There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training](https://mlsanity.com/paper/2510.12586/) (2025)
- [Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces](https://mlsanity.com/paper/2506.07903/) (2025)
- [Unified Latents (UL): How to train your latents](https://mlsanity.com/paper/2602.17270/) (2026)
- [FARMER: Flow AutoRegressive Transformer over Pixels](https://mlsanity.com/paper/2510.23588/) (2025)
- [Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion](https://mlsanity.com/paper/2512.04926/) (2025)
- [Guiding a Diffusion Transformer with the Internal Dynamics of Itself](https://mlsanity.com/paper/2512.24176/) (2025)
- [Guidance in the Frequency Domain Enables High-Fidelity Sampling at Low CFG Scales](https://mlsanity.com/paper/2506.19713/) (2025)
- [Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoders](https://mlsanity.com/paper/2602.10099/) (2026)
- [Improving Classifier-Free Guidance in Masked Diffusion: Low-Dim Theoretical Insights with High-Dim Impact](https://mlsanity.com/paper/2507.08965/) (2025)
- [D-AR: Diffusion via Autoregressive Models](https://mlsanity.com/paper/2505.23660/) (2025)
- [Autoregressive Image Generation with Masked Bit Modeling](https://mlsanity.com/paper/2602.09024/) (2026)
- [L2P: Unlocking Latent Potential for Pixel Generation](https://mlsanity.com/paper/2605.12013/) (2026)
- [FLEX: A Backbone for Diffusion-Based Modeling of Spatio-temporal Physical Systems](https://mlsanity.com/paper/2505.17351/) (2025)
- [Image Generation with a Sphere Encoder](https://mlsanity.com/paper/2602.15030/) (2026)
- [Asymmetric Flow Models](https://mlsanity.com/paper/2605.12964/) (2026)
- [Stochastic Interpolants via Conditional Dependent Coupling](https://mlsanity.com/paper/2509.23122/) (2025)

## Abstract (arXiv)

Latent diffusion models have become the popular choice for scaling up diffusion models for high resolution image synthesis. Compared to pixel-space models that are trained end-to-end, latent models are perceived to be more efficient and to produce higher image quality at high resolution. Here we challenge these notions, and show that pixel-space models can be very competitive to latent models both in quality and efficiency, achieving 1.5 FID on ImageNet512 and new SOTA results on ImageNet128, ImageNet256 and Kinetics600. We present a simple recipe for scaling end-to-end pixel-space diffusion models to high resolutions. 1: Use the sigmoid loss-weighting (Kingma & Gao, 2023) with our prescribed hyper-parameters. 2: Use our simplified memory-efficient architecture with fewer skip-connections. 3: Scale the model to favor processing the image at a high resolution with fewer parameters, rather than using more parameters at a lower resolution. Combining these with guidance intervals, we obtain a family of pixel-space diffusion models we call Simpler Diffusion (SiD2).