# LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation arXiv: https://arxiv.org/abs/2311.00353 ยท HTML: https://mlsanity.com/paper/2311.00353/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 21 of its 24 references that are in the map. - First posted on arXiv 1 Nov 2023. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 21 of 21) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022) - [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020) - [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022) - [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023) - [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023) - [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023) - [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023) - [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023) - [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022) - [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023) - [VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet](https://mlsanity.com/paper/2307.14073/) (2023) ## Built on by (most cited first, 12 of 12) - [SkyReels-A2: Compose Anything in Video Diffusion Transformers](https://mlsanity.com/paper/2504.02436/) (2025) - [MotionCraft: Physics-based Zero-Shot Video Generation](https://mlsanity.com/paper/2405.13557/) (2024) - [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024) - [SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers](https://mlsanity.com/paper/2502.10841/) (2025) - [Ingredients: Blending Custom Photos with Video Diffusion Transformers](https://mlsanity.com/paper/2501.01790/) (2025) - [MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesis](https://mlsanity.com/paper/2410.20974/) (2024) - [Video Diffusion Transformers are In-Context Learners](https://mlsanity.com/paper/2412.10783/) (2024) - [Morpheus: Text-Driven 3D Gaussian Splat Shape and Color Stylization](https://mlsanity.com/paper/2503.02009/) (2025) - [FreeInv: Free Lunch for Improving DDIM Inversion](https://mlsanity.com/paper/2503.23035/) (2025) - [Motion Attribution for Video Generation](https://mlsanity.com/paper/2601.08828/) (2026) - [FloAt: Flow Warping of Self-Attention for Clothing Animation Generation](https://mlsanity.com/paper/2411.15028/) (2024) - [Improving Temporal Consistency and Fidelity at Inference-time in Perceptual Video Restoration by Zero-shot Image-based Diffusion Models](https://mlsanity.com/paper/2510.25420/) (2025) ## Abstract (arXiv) Leveraging the generative ability of image diffusion models offers great potential for zero-shot video-to-video translation. The key lies in how to maintain temporal consistency across generated video frames by image diffusion models. Previous methods typically adopt cross-frame attention, i.e., sharing the textitkey and textitvalue tokens across attentions of different frames, to encourage the temporal consistency. However, in those works, temporal inconsistency issue may not be thoroughly solved, rendering the fidelity of generated videos limited.%The current state of the art cross-frame attention method aims at maintaining fine-grained visual details across frames, but it is still challenged by the temporal coherence problem. In this paper, we find the bottleneck lies in the unconstrained query tokens and propose a new zero-shot video-to-video translation framework, named textitLatentWarp. Our approach is simple: to constrain the query tokens to be temporally consistent, we further incorporate a warping operation in the latent space to constrain the query tokens. Specifically, based on the optical flow obtained from the original video, we warp the generated latent features of last frame to align with the current frame during the denoising process. As a result, the corresponding regions across the adjacent frames can share closely-related query tokens and attention outputs, which can further improve latent-level consistency to enhance visual temporal coherence of generated videos. Extensive experiment results demonstrate the superiority of textitLatentWarp in achieving video-to-video translation with temporal coherence.