# Generative Video Propagation

arXiv: https://arxiv.org/abs/2412.19761 · HTML: https://mlsanity.com/paper/2412.19761/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 61 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [MotionDirector: Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2310.08465/) (2023)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)

## Built on by (most cited first, 30 of 36)

- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation](https://mlsanity.com/paper/2502.04299/) (2025)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation](https://mlsanity.com/paper/2503.14350/) (2025)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment](https://mlsanity.com/paper/2509.17818/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [Region-Constraint In-Context Generation for Instructional Video Editing](https://mlsanity.com/paper/2512.17650/) (2025)
- [OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions](https://mlsanity.com/paper/2506.23361/) (2025)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [DreamVE: Unified Instruction-based Image and Video Editing](https://mlsanity.com/paper/2508.06080/) (2025)
- [O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing](https://mlsanity.com/paper/2509.01596/) (2025)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing](https://mlsanity.com/paper/2512.06065/) (2025)
- [VideoCoF: Unified Video Editing with Temporal Reasoner](https://mlsanity.com/paper/2512.07469/) (2025)
- [SeqTex: Generate Mesh Textures in Video Sequence](https://mlsanity.com/paper/2507.04285/) (2025)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Object-WIPER : Training-Free Object and Associated Effect Removal in Videos](https://mlsanity.com/paper/2601.06391/) (2026)
- [Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformer](https://mlsanity.com/paper/2508.09131/) (2025)
- [FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing](https://mlsanity.com/paper/2601.01720/) (2026)
- [EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing](https://mlsanity.com/paper/2602.15031/) (2026)
- [SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation](https://mlsanity.com/paper/2603.16864/) (2026)
- [LazyDrag: Enabling Stable Drag-Based Editing on Multi-Modal Diffusion Transformers via Explicit Correspondence](https://mlsanity.com/paper/2509.12203/) (2025)
- [V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties](https://mlsanity.com/paper/2512.11799/) (2025)
- [NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing](https://mlsanity.com/paper/2603.02802/) (2026)
- [Tuning-free Visual Effect Transfer across Videos](https://mlsanity.com/paper/2601.07833/) (2026)
- [TRACE: Object Motion Editing in Videos with First-Frame Trajectory Guidance](https://mlsanity.com/paper/2603.25707/) (2026)
- [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026)

## Abstract (arXiv)

Large-scale video generation models have the inherent ability to realistically model natural scenes. In this paper, we demonstrate that through a careful design of a generative video propagation framework, various video tasks can be addressed in a unified way by leveraging the generative power of such models. Specifically, our framework, GenProp, encodes the original video with a selective content encoder and propagates the changes made to the first frame using an image-to-video generation model. We propose a data generation scheme to cover multiple video tasks based on instance-level video segmentation datasets. Our model is trained by incorporating a mask prediction decoder head and optimizing a region-aware loss to aid the encoder to preserve the original content while the generation model propagates the modified region. This novel design opens up new possibilities: In editing scenarios, GenProp allows substantial changes to an object's shape; for insertion, the inserted objects can exhibit independent motion; for removal, GenProp effectively removes effects like shadows and reflections from the whole video; for tracking, GenProp is capable of tracking objects and their associated effects together. Experiment results demonstrate the leading performance of our model in various video tasks, and we further provide in-depth analyses of the proposed framework.