mlsanityData to 14 Oct 2026

Looking Backward: Streaming Video-to-Video Translation with Feature Banks

arXiv 2405.15757 ↗First posted 24 May 2024cs.CV cs.MMcited by 38

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper introduces StreamV2V, a diffusion model that achieves real-time streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V methods using batches to process limited frames, we opt to process frames in a streaming fashion, to support unlimited frames. At the heart of StreamV2V lies a backward-looking principle that relates the present to the past. This is realized by maintaining a feature bank, which archives information from past frames. For incoming frames, StreamV2V extends self-attention to include banked keys and values and directly fuses similar past features into the output. The feature bank is continually updated by merging stored and new features, making it compact but informative. StreamV2V stands out for its adaptability and efficiency, seamlessly integrating with image diffusion models without fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative metrics and user studies confirm StreamV2V's exceptional ability to maintain temporal consistency.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 56 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

48 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation2 May
  2. 2024AnimateDiff-Lightning: Cross-Model Diffusion Distillation19 Mar
  3. 2024Video Editing via Factorized Diffusion Distillation14 Mar
  4. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis29 Dec
  5. 2023Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis20 Dec
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  7. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  8. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  9. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  10. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  11. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  12. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
1 more
  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  11. 2023Consistency Modelscited by 1,822
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
23 more
  1. 2022Token Merging: Your ViT But Fastercited by 1,021
  2. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  4. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  5. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  6. 2023Adversarial Diffusion Distillationcited by 759
  7. 2022On Distillation of Guided Diffusion Modelscited by 725
  8. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  9. 2023Emergent Correspondence from Image Diffusioncited by 446
  10. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  11. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  12. 2018Learning Blind Video Temporal Consistencycited by 253
  13. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  14. 2023Token Merging for Fast Stable Diffusioncited by 224
  15. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  16. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  17. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 213
  18. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  19. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Cachingcited by 151
  20. 2024Training-Free Consistent Text-to-Image Generationcited by 125
  21. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  22. 2020Actor-Context-Actor Relation Network for Spatio-Temporal Action Localizationcited by 91
  23. 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 21.

38 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  3. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  4. 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generationcited by 30
  5. 2025Human Motion Video Generation: A Surveycited by 25
  6. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  7. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
  8. 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
  9. 2026Generative World Renderercited by 10
  10. 2025OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Modelscited by 9
  11. 2025InfVSR: Toward Consistency-Driven Streaming Generative Video Super-Resolutioncited by 8
  12. 2025EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editingcited by 8
26 more
  1. 2026StreamingEffect: Real-Time Human-Centric Video Effect Generationcited by 8
  2. 2026GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generatorcited by 7
  3. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 7
  4. 2026LiveLight: Real-time Streaming Video Relighting with Interactive Controlcited by 7
  5. 2024MatchDiffusion: Training-free Generation of Match-cutscited by 5
  6. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
  7. 2026ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translationcited by 3
  8. 2026ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editingcited by 3
  9. 2026In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusioncited by 3
  10. 2024Ada-VE: Training-Free Consistent Video Editing Using Adaptive Motion Priorcited by 2
  11. 2024RAIN: Real-time Animation of Infinite Video Streamcited by 2
  12. 2025ObjectAlign: Neuro-Symbolic Object Consistency Verification and Correctioncited by 2
  13. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformerscited by 2
  14. 2026StreamEdit: Training-Free Video Editing via Few-Step Streaming Video Generationcited by 2
  15. 2026EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editingcited by 2
  16. 2025AnchorSync: Global Consistency Optimization for Long Video Editingcited by 1
  17. 2026Video Compression Meets Video Generation: Latent Inter-Frame Pruning with Attention Recoverycited by 1
  18. 2026CityRAG: Stepping Into a City via Spatially-Grounded Video Generationcited by 1
  19. 2025Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioningcited by 0
  20. 2025PipeFlow: Pipelined Processing and Motion-Aware Frame Selection for Long-Form Video Editingcited by 0
  21. 2026Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPUcited by 0
  22. 2026Importance-Aware OBS Pruning for Diffusion Modelscited by 0
  23. 2026Unified Video Dense Prediction from Disjoint Datacited by 0
  24. 2026EditaLive! Unified Character Video Editing for Live Streamingcited by 0
  25. 2026Accelerating Video Diffusion via Training-Free Trajectory Routingcited by 0
  26. 2026No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Live2Diff: Live Stream Translation via Uni-directional Attention in Video Diffusion Models
  2. 2026Rethinking Streaming Video Diffusion Model: Context, Execution, and Training
  3. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  4. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  5. 2026VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding
  6. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text
  7. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
  8. 2025StreamDiT: Real-Time Streaming Text-to-Video Generation