# StableV2V: Stablizing Shape Consistency in Video-to-Video Editing

arXiv: https://arxiv.org/abs/2411.11045 · HTML: https://mlsanity.com/paper/2411.11045/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 78 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 30 of 60)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)

## Built on by (most cited first, 26 of 26)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment](https://mlsanity.com/paper/2510.11647/) (2025)
- [FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editing](https://mlsanity.com/paper/2506.05046/) (2025)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding](https://mlsanity.com/paper/2504.10825/) (2025)
- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [DreamVE: Unified Instruction-based Image and Video Editing](https://mlsanity.com/paper/2508.06080/) (2025)
- [O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing](https://mlsanity.com/paper/2509.01596/) (2025)
- [Consistent Video Editing as Flow-Driven Image-to-Video Generation](https://mlsanity.com/paper/2506.07713/) (2025)
- [SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing](https://mlsanity.com/paper/2605.25193/) (2026)
- [Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing](https://mlsanity.com/paper/2606.30599/) (2026)
- [FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing](https://mlsanity.com/paper/2601.01720/) (2026)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [DFVEdit: Conditional Delta Flow Vector for Zero-shot Video Editing](https://mlsanity.com/paper/2506.20967/) (2025)
- [NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing](https://mlsanity.com/paper/2603.02802/) (2026)
- [LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention](https://mlsanity.com/paper/2605.04569/) (2026)
- [ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships](https://mlsanity.com/paper/2607.14681/) (2026)
- [ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editing](https://mlsanity.com/paper/2607.21529/) (2026)
- [CoT-Edit: Let CoT Guide Instruction Video Editing](https://mlsanity.com/paper/2608.01113/) (2026)
- [Controllable Video Generation with Provable Disentanglement](https://mlsanity.com/paper/2502.02690/) (2025)
- [MoMaps: Semantics-Aware Scene Motion Generation with Motion Maps](https://mlsanity.com/paper/2510.11107/) (2025)
- [Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editing](https://mlsanity.com/paper/2505.23134/) (2025)
- [Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis](https://mlsanity.com/paper/2606.29020/) (2026)
- [QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing](https://mlsanity.com/paper/2610.17829/) (2026)

## Abstract (arXiv)

Recent advancements of generative AI have significantly promoted content creation and editing, where prevailing studies further extend this exciting progress to video editing. In doing so, these studies mainly transfer the inherent motion patterns from the source videos to the edited ones, where results with inferior consistency to user prompts are often observed, due to the lack of particular alignments between the delivered motions and edited contents. To address this limitation, we present a shape-consistent video editing method, namely StableV2V, in this paper. Our method decomposes the entire editing pipeline into several sequential procedures, where it edits the first video frame, then establishes an alignment between the delivered motions and user prompts, and eventually propagates the edited contents to all other frames based on such alignment. Furthermore, we curate a testing benchmark, namely DAVIS-Edit, for a comprehensive evaluation of video editing, considering various types of prompts and difficulties. Experimental results and analyses illustrate the outperforming performance, visual consistency, and inference efficiency of our method compared to existing state-of-the-art studies.