mlsanityData to 7 Oct 2026

SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions

arXiv 2403.16627 ↗First posted 25 Mar 2024cs.CVcited by 23

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent advancements in diffusion models have positioned them at the forefront of image generation. Despite their superior performance, diffusion models are not without drawbacks; they are characterized by complex architectures and substantial computational demands, resulting in significant latency due to their iterative sampling process. To mitigate these limitations, we introduce a dual approach involving model miniaturization and a reduction in sampling steps, aimed at significantly decreasing model latency. Our methodology leverages knowledge distillation to streamline the U-Net and image decoder architectures, and introduces an innovative one-step DM training technique that utilizes feature matching and score distillation. We present two models, SDXS-512 and SDXS-1024, achieving inference speeds of approximately 100 FPS (30x faster than SD v1.5) and 30 FPS (60x faster than SDXL) on a single GPU, respectively. Moreover, our training approach offers promising applications in image-conditioned control, facilitating efficient image-to-image translation.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 56 of its 62 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

56 of 62 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillation21 Feb
  2. 2024Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Loss5 Jan
  3. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  4. 2023Adversarial Diffusion Distillation28 Nov
  5. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Module9 Nov
  6. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  8. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models29 May
  9. 2023Structural Pruning for Diffusion Models18 May
  10. 2023PTQD: Accurate Post-Training Quantization for Diffusion Models18 May
  11. 2023Consistency Models2 Mar
  12. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models9 Feb
11 more
  1. 2023Q-Diffusion: Quantizing Diffusion Models8 Feb
  2. 2022Post-training Quantization on Diffusion Models28 Nov
  3. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models2 Nov
  4. 2022On Distillation of Guided Diffusion Models6 Oct
  5. 2022Classifier-Free Diffusion Guidance26 Jul
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  7. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  10. 2020Denoising Diffusion Implicit Models6 Oct
  11. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
21 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  10. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  11. 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 792
  12. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  13. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  14. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  15. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 360
  16. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  17. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  18. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  19. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  20. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  21. 2024Score Distillation Sampling with Learned Manifold Correctivecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 7; 2026: 3.

23 papers.

Sort:

By area:

  1. 2024One-Step Diffusion Distillation through Score Implicit Matchingcited by 68
  2. 2024AddSR: Accelerating Diffusion-based Blind Super-Resolution with Adversarial Diffusion Distillationcited by 66
  3. 2024Adversarial Diffusion Compression for Real-World Image Super-Resolutioncited by 49
  4. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferencescited by 30
  5. 2024SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Trainingcited by 27
  6. 2024Flow Generator Matchingcited by 26
  7. 2024FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Modelscited by 21
  8. 2024BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Modelscited by 17
  9. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  10. 2025Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instructioncited by 13
  11. 2024Towards Understanding and Quantifying Uncertainty for Text-to-Image Generationcited by 11
  12. 2025M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversioncited by 10
11 more
  1. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-trainingcited by 8
  2. 2025Adding Additional Control to One-Step Diffusion with Joint Distribution Matchingcited by 8
  3. 2025Scaling Down Text Encoders of Text-to-Image Diffusion Modelscited by 8
  4. 2024VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversarycited by 6
  5. 2025TS-DP: Reinforcement Speculative Decoding For Temporal Adaptive Diffusion Policy Accelerationcited by 6
  6. 2025LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Groundingcited by 5
  7. 2025Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controlscited by 3
  8. 2024Class-Conditional self-reward mechanism for improved Text-to-Image modelscited by 0
  9. 2026Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultracited by 0
  10. 2026LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Modelscited by 0
  11. 2026Accelerating Video Diffusion via Training-Free Trajectory Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
  2. 2024Plug-and-Play Diffusion Distillation
  3. 2025DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference
  4. 2025Toward Lightweight and Fast Decoders for Diffusion Models in Image and Video Generation
  5. 2023Diffusion Models Without Attention
  6. 2025Glance: Accelerating Diffusion Models with 1 Sample
  7. 2026One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration
  8. 2026LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation