mlsanityData to 7 Oct 2026

FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation

arXiv 2403.12962 ↗First posted 19 Mar 2024cs.CVcited by 48

On the map: Diffusion models → Video generation

Abstract (arXiv)

The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 47 of its 52 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

47 of 52 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  2. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models2 Oct
  3. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Models28 Sep
  4. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  5. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  7. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  8. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  9. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  11. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
7 more
  1. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  2. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  3. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  6. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
16 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  4. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  5. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  7. 2023Zero-shot Image-to-Image Translationcited by 503
  8. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  9. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  10. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  11. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  12. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  13. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  14. 2023TaleCrafter: Interactive Story Visualization with Multiple Characterscited by 71
  15. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 65
  16. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 18; 2026: 12.

48 papers.

Sort:

By area:

  1. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  2. 2025Controllable Video Generation: A Surveycited by 93
  3. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  4. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  5. 2024DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Modelscited by 33
  6. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  7. 2024TextToon: Real-Time Text Toonify Head Avatar from Single Videocited by 21
  8. 2025Generative AI for Cel-Animation: A Surveycited by 17
  9. 2024Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observationcited by 16
  10. 2025Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Spacecited by 14
  11. 2024UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Modelcited by 13
  12. 2024MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesiscited by 12
36 more
  1. 2025Controllable Weather Synthesis and Removal with Video Diffusion Modelscited by 11
  2. 2024VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessmentcited by 10
  3. 2025TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMscited by 10
  4. 2024MVOC: a training-free multiple video object composition method with diffusion modelscited by 8
  5. 2024Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Modelcited by 8
  6. 2024GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generationcited by 8
  7. 2026Generative World Renderercited by 8
  8. 2025Understanding Attention Mechanism in Video Diffusion Modelscited by 7
  9. 2025Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusioncited by 7
  10. 2026Evolution of Video Generative Foundationscited by 6
  11. 2025AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selectioncited by 5
  12. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 5
  13. 2026Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Drivingcited by 5
  14. 2024RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidancecited by 4
  15. 2025PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animationcited by 4
  16. 2025ANYPORTAL: Zero-Shot Consistent Video Background Replacementcited by 4
  17. 2025PickStyle: Video-to-Video Style Transfer with Context-Style Adapterscited by 3
  18. 2026Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Glovescited by 3
  19. 2024VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editingcited by 2
  20. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  21. 2024MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulationcited by 2
  22. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  23. 2026WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformationcited by 2
  24. 2025Social Human Robot Embodied Conversation (SHREC) Dataset: Benchmarking Foundational Models' Social Reasoningcited by 1
  25. 2025Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editingcited by 1
  26. 2025Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesiscited by 1
  27. 2026DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformercited by 1
  28. 2024PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with Benchmark Datasetcited by 0
  29. 2025StgcDiff: Spatial-Temporal Graph Condition Diffusion for Sign Language Transition Generationcited by 0
  30. 2025Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondencecited by 0
  31. 2026Beyond Consistency: Preserving Temporal Structure in Zero-Shot Video Editingcited by 0
  32. 2026TeleMorpher: Toward Robust Simultaneous Motion-Location Editingcited by 0
  33. 2026Event Stream based Multi-Modal Video Anomaly Detection: A Benchmark Dataset and Algorithmscited by 0
  34. 2026Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generationcited by 0
  35. 2026OSVE: One Step Video Editing with One Step Diffusion Modelscited by 0
  36. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation
  2. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  3. 2025Emergent Temporal Correspondences from Video Diffusion Transformers
  4. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  5. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  6. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers
  7. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  8. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer