# VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer

arXiv: https://arxiv.org/abs/2502.05979 · HTML: https://mlsanity.com/paper/2502.05979/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [DreamVideo: Composing Your Dream Videos with Customized Subject and Motion](https://mlsanity.com/paper/2312.04433/) (2023)
- [MotionDirector: Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2310.08465/) (2023)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [MotionBooth: Motion-Aware Customized Text-to-Video Generation](https://mlsanity.com/paper/2406.17758/) (2024)
- [VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2312.00845/) (2023)
- [LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation](https://mlsanity.com/paper/2310.10769/) (2023)
- [Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2402.14780/) (2024)
- [I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models](https://mlsanity.com/paper/2405.16537/) (2024)
- [Towards Understanding Mixture of Experts in Deep Learning](https://mlsanity.com/paper/2208.02813/) (2022)
- [MotionCrafter: One-Shot Motion Customization of Diffusion Models](https://mlsanity.com/paper/2312.05288/) (2023)

## Built on by (most cited first, 16 of 16)

- [Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation](https://mlsanity.com/paper/2508.07981/) (2025)
- [Video-As-Prompt: Unified Semantic Control for Video Generation](https://mlsanity.com/paper/2510.20888/) (2025)
- [VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learning](https://mlsanity.com/paper/2510.25772/) (2025)
- [GenCompositor: Generative Video Compositing with Diffusion Transformer](https://mlsanity.com/paper/2509.02460/) (2025)
- [StreamingEffect: Real-Time Human-Centric Video Effect Generation](https://mlsanity.com/paper/2605.17019/) (2026)
- [OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer](https://mlsanity.com/paper/2601.14250/) (2026)
- [View-Consistent Diffusion Representations for 3D-Consistent Video Generation](https://mlsanity.com/paper/2511.18991/) (2025)
- [IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning](https://mlsanity.com/paper/2512.15635/) (2025)
- [Over++: Generative Video Compositing for Layer Interaction Effects](https://mlsanity.com/paper/2512.19661/) (2025)
- [AesRM: Improving Video Aesthetics with Expert-Level Feedback](https://mlsanity.com/paper/2604.28078/) (2026)
- [Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video](https://mlsanity.com/paper/2601.22013/) (2026)
- [EffectMaker: Unifying Reasoning and Generation for Customized Visual Effect Creation](https://mlsanity.com/paper/2603.06014/) (2026)
- [EasyVFX: Frequency-Driven Decoupling for Resource-Efficient VFX Generation](https://mlsanity.com/paper/2605.22051/) (2026)
- [Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA](https://mlsanity.com/paper/2603.08210/) (2026)
- [P-Flow: Prompting Visual Effects Generation](https://mlsanity.com/paper/2603.22091/) (2026)
- [Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generation](https://mlsanity.com/paper/2606.13971/) (2026)

## Abstract (arXiv)

Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.