# VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet

arXiv: https://arxiv.org/abs/2307.14073 · HTML: https://mlsanity.com/paper/2307.14073/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)

## Built on by (most cited first, 30 of 83)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)
- [FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2312.17681/) (2023)
- [Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model](https://mlsanity.com/paper/2404.09967/) (2024)
- [VideoStudio: Generating Consistent-Content and Multi-Scene Videos](https://mlsanity.com/paper/2401.01256/) (2024)
- [Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models](https://mlsanity.com/paper/2310.01107/) (2023)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation](https://mlsanity.com/paper/2403.12962/) (2024)
- [FullDiT: Multi-Task Video Generative Foundation Model with Full Attention](https://mlsanity.com/paper/2503.19907/) (2025)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation](https://mlsanity.com/paper/2411.08380/) (2024)
- [AVID: Adapting Video Diffusion Models to World Models](https://mlsanity.com/paper/2410.12822/) (2024)
- [DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing](https://mlsanity.com/paper/2403.12002/) (2024)
- [OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding](https://mlsanity.com/paper/2504.10825/) (2025)
- [HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness](https://mlsanity.com/paper/2406.07754/) (2024)
- [Spectral Motion Alignment for Video Motion Transfer using Diffusion Models](https://mlsanity.com/paper/2403.15249/) (2024)
- [Fine-grained Controllable Video Generation via Object Appearance and Context](https://mlsanity.com/paper/2312.02919/) (2023)
- [EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing](https://mlsanity.com/paper/2403.16111/) (2024)
- [DIVE: Taming DINO for Subject-Driven Video Editing](https://mlsanity.com/paper/2412.03347/) (2024)
- [OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Models](https://mlsanity.com/paper/2411.10501/) (2024)
- [A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing](https://mlsanity.com/paper/2312.05856/) (2023)
- [UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control](https://mlsanity.com/paper/2403.02332/) (2024)
- [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://mlsanity.com/paper/2504.16081/) (2025)
- [Video-As-Prompt: Unified Semantic Control for Video Generation](https://mlsanity.com/paper/2510.20888/) (2025)
- [MotionV2V: Editing Motion in a Video](https://mlsanity.com/paper/2511.20640/) (2025)
- [Intention-driven Ego-to-Exo Video Generation](https://mlsanity.com/paper/2403.09194/) (2024)
- [LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation](https://mlsanity.com/paper/2311.00353/) (2023)

## Abstract (arXiv)

Recently, diffusion models like StableDiffusion have achieved impressive image generation results. However, the generation process of such diffusion models is uncontrollable, which makes it hard to generate videos with continuous and consistent content. In this work, by using the diffusion model with ControlNet, we proposed a new motion-guided video-to-video translation framework called VideoControlNet to generate various videos based on the given prompts and the condition from the input video. Inspired by the video codecs that use motion information for reducing temporal redundancy, our framework uses motion information to prevent the regeneration of the redundant areas for content consistency. Specifically, we generate the first frame (i.e., the I-frame) by using the diffusion model with ControlNet. Then we generate other key frames (i.e., the P-frame) based on the previous I/P-frame by using our newly proposed motion-guided P-frame generation (MgPG) method, in which the P-frames are generated based on the motion information and the occlusion areas are inpainted by using the diffusion model. Finally, the rest frames (i.e., the B-frame) are generated by using our motion-guided B-frame interpolation (MgBI) module. Our experiments demonstrate that our proposed VideoControlNet inherits the generation capability of the pre-trained large diffusion model and extends the image diffusion model to the video diffusion model by using motion information. More results are provided at our project page.