# MAVIN: Multi-Action Video Generation with Diffusion Models via Transition Video Infilling arXiv: https://arxiv.org/abs/2405.18003 ยท HTML: https://mlsanity.com/paper/2405.18003/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 30 of its 65 references that are in the map. - First posted on arXiv 28 May 2024. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 30 of 30) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017) - [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022) - [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023) - [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021) - [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023) - [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023) - [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022) - [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022) - [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023) - [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023) - [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023) - [StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text](https://mlsanity.com/paper/2403.14773/) (2024) - [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024) - [On the Importance of Noise Scheduling for Diffusion Models](https://mlsanity.com/paper/2301.10972/) (2023) - [DreamVideo: Composing Your Dream Videos with Customized Subject and Motion](https://mlsanity.com/paper/2312.04433/) (2023) - [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023) - [Diffusion Models for Video Prediction and Infilling](https://mlsanity.com/paper/2206.07696/) (2022) - [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://mlsanity.com/paper/2402.04324/) (2024) - [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://mlsanity.com/paper/2402.03161/) (2024) - [FreeInit: Bridging Initialization Gap in Video Diffusion Models](https://mlsanity.com/paper/2312.07537/) (2023) - [VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2312.00845/) (2023) - [Video Interpolation with Diffusion Models](https://mlsanity.com/paper/2404.01203/) (2024) - [LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation](https://mlsanity.com/paper/2310.10769/) (2023) - [Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts](https://mlsanity.com/paper/2403.08268/) (2024) - [LivePhoto: Real Image Animation with Text-guided Motion Control](https://mlsanity.com/paper/2312.02928/) (2023) - [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024) - [AtomoVideo: High Fidelity Image-to-Video Generation](https://mlsanity.com/paper/2403.01800/) (2024) - [Pix2Gif: Motion-Guided Diffusion for GIF Generation](https://mlsanity.com/paper/2403.04634/) (2024) ## Built on by (most cited first, 4 of 4) - [Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesis](https://mlsanity.com/paper/2410.07155/) (2024) - [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024) - [TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generation](https://mlsanity.com/paper/2604.19473/) (2026) - [No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generation](https://mlsanity.com/paper/2609.38691/) (2026) ## Abstract (arXiv) Diffusion-based video generation has achieved significant progress, yet generating multiple actions that occur sequentially remains a formidable task. Directly generating a video with sequential actions can be extremely challenging due to the scarcity of fine-grained action annotations and the difficulty in establishing temporal semantic correspondences and maintaining long-term consistency. To tackle this, we propose an intuitive and straightforward solution: splicing multiple single-action video segments sequentially. The core challenge lies in generating smooth and natural transitions between these segments given the inherent complexity and variability of action transitions. We introduce MAVIN (Multi-Action Video INfilling model), designed to generate transition videos that seamlessly connect two given videos, forming a cohesive integrated sequence. MAVIN incorporates several innovative techniques to address challenges in the transition video infilling task. Firstly, a consecutive noising strategy coupled with variable-length sampling is employed to handle large infilling gaps and varied generation lengths. Secondly, boundary frame guidance (BFG) is proposed to address the lack of semantic guidance during transition generation. Lastly, a Gaussian filter mixer (GFM) dynamically manages noise initialization during inference, mitigating train-test discrepancy while preserving generation flexibility. Additionally, we introduce a new metric, CLIP-RS (CLIP Relative Smoothness), to evaluate temporal coherence and smoothness, complementing traditional quality-based metrics. Experimental results on horse and tiger scenarios demonstrate MAVIN's superior performance in generating smooth and coherent video transitions compared to existing methods.