mlsanityData to 13 Oct 2026

Streaming Video Diffusion: Online Video Editing with Diffusion Models

arXiv 2405.19726 ↗First posted 30 May 2024cs.CVcited by 13

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present a novel task called online video editing, which is designed to edit streaming frames while maintaining temporal consistency. Unlike existing offline video editing assuming all frames are pre-established and accessible, online video editing is tailored to real-life applications such as live streaming and online chat, requiring (1) fast continual step inference, (2) long-term temporal modeling, and (3) zero-shot video editing capability. To solve these issues, we propose Streaming Video Diffusion (SVDiff), which incorporates the compact spatial-aware temporal recurrence into off-the-shelf Stable Diffusion and is trained with the segment-level scheme on large-scale long videos. This simple yet effective setup allows us to obtain a single model that is capable of executing a broad range of videos and editing each streaming frame with temporal coherence. Our experiments indicate that our model can edit long, high-quality videos with remarkable results, achieving a real-time inference speed of 15.2 FPS at a resolution of 512x512.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 36 of its 40 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

36 of 40 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  2. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  3. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Flexible Diffusion Modeling of Long Videos23 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  4. 2023Visual Instruction Tuningcited by 7,827
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  6. 2022Classifier-Free Diffusion Guidancecited by 5,785
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,323
  8. 2020Longformer: The Long-Document Transformercited by 3,393
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,421
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,732
14 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  2. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 837
  3. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 619
  4. 2018Video-to-Video Synthesiscited by 390
  5. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  6. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  7. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  8. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  9. 2022Recurrent Memory Transformercited by 199
  10. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  11. 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 57
  12. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
  13. 2023Streaming Video Modelcited by 10
  14. 2022Class-attention Video Transformer for Engagement Intensity Predictioncited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 590
  2. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 414
  3. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 30
  4. 2026AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoisingcited by 6
  5. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 6
  6. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
  7. 2026In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusioncited by 3
  8. 2025Zero-Shot Video Deraining with Video Diffusion Modelscited by 2
  9. 2025SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editingcited by 1
  10. 2026RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editingcited by 1
  11. 2026EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editingcited by 1
  12. 2026Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPUcited by 0
1 more
  1. 2026EditaLive! Unified Character Video Editing for Live Streamingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Streaming Video Editing with Easy Adaptation
  2. 2024Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models
  3. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models
  4. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
  5. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  6. 2025Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework
  7. 2026Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model
  8. 2026Rethinking Streaming Video Diffusion Model: Context, Execution, and Training