# StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation

arXiv: https://arxiv.org/abs/2312.12491 · HTML: https://mlsanity.com/paper/2312.12491/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021)
- [LCM-LoRA: A Universal Stable-Diffusion Acceleration Module](https://mlsanity.com/paper/2311.05556/) (2023)
- [Fast Sampling of Diffusion Models via Operator Learning](https://mlsanity.com/paper/2211.13449/) (2022)
- [Learning to Efficiently Sample from Diffusion Probabilistic Models](https://mlsanity.com/paper/2106.03802/) (2021)

## Built on by (most cited first, 30 of 58)

- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2506.09350/) (2025)
- [The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control](https://mlsanity.com/paper/2412.03568/) (2024)
- [Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing](https://mlsanity.com/paper/2411.16375/) (2024)
- [AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion](https://mlsanity.com/paper/2503.07418/) (2025)
- [StreamDiT: Real-Time Streaming Text-to-Video Generation](https://mlsanity.com/paper/2507.03745/) (2025)
- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [Looking Backward: Streaming Video-to-Video Translation with Feature Banks](https://mlsanity.com/paper/2405.15757/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching](https://mlsanity.com/paper/2402.14167/) (2024)
- [StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation](https://mlsanity.com/paper/2511.07399/) (2025)
- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Generative Image as Action Models](https://mlsanity.com/paper/2407.07875/) (2024)
- [Generative AI for Film Creation: A Survey of Recent Advances](https://mlsanity.com/paper/2504.08296/) (2025)
- [FloodDiffusion: Tailored Diffusion Forcing for Streaming Motion Generation](https://mlsanity.com/paper/2512.03520/) (2025)
- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [Immiscible Diffusion: Accelerating Diffusion Training with Noise Assignment](https://mlsanity.com/paper/2406.12303/) (2024)
- [Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy](https://mlsanity.com/paper/2410.09873/) (2024)
- [Streaming Video Diffusion: Online Video Editing with Diffusion Models](https://mlsanity.com/paper/2405.19726/) (2024)
- [Text-to-Image Synthesis: A Decade Survey](https://mlsanity.com/paper/2411.16164/) (2024)
- [MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation](https://mlsanity.com/paper/2511.22989/) (2025)
- [EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing](https://mlsanity.com/paper/2512.06065/) (2025)
- [SemanticDraw: Towards Real-Time Interactive Content Creation from Image Diffusion Models](https://mlsanity.com/paper/2403.09055/) (2024)
- [Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models](https://mlsanity.com/paper/2407.08701/) (2024)
- [LiveLight: Real-time Streaming Video Relighting with Interactive Control](https://mlsanity.com/paper/2608.01771/) (2026)
- [SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches](https://mlsanity.com/paper/2502.07556/) (2025)
- [Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment](https://mlsanity.com/paper/2508.06160/) (2025)
- [Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!](https://mlsanity.com/paper/2510.03550/) (2025)

## Abstract (arXiv)

We introduce StreamDiffusion, a real-time diffusion pipeline designed for interactive image generation. Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction. This limitation becomes particularly evident in scenarios involving continuous input, such as Metaverse, live video streaming, and broadcasting, where high throughput is imperative. To address this, we present a novel approach that transforms the original sequential denoising into the batching denoising process. Stream Batch eliminates the conventional wait-and-interact approach and enables fluid and high throughput streams. To handle the frequency disparity between data input and model throughput, we design a novel input-output queue for parallelizing the streaming process. Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG), which requires additional U-Net computation. To mitigate the redundant computations, we propose a novel residual classifier-free guidance (RCFG) algorithm that reduces the number of negative conditional denoising steps to only one or even zero. Besides, we introduce a stochastic similarity filter(SSF) to optimize power consumption. Our Stream Batch achieves around 1.5x speedup compared to the sequential denoising method at different denoising levels. The proposed RCFG leads to speeds up to 2.05x higher than the conventional CFG. Combining the proposed strategies and existing mature acceleration tools makes the image-to-image generation achieve up-to 91.07fps on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers over 59.56x. Furthermore, our proposed StreamDiffusion also significantly reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one RTX4090, respectively.