# Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation

arXiv: https://arxiv.org/abs/2508.07981 · HTML: https://mlsanity.com/paper/2508.07981/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 63 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 30 of 52)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer](https://mlsanity.com/paper/2504.20690/) (2025)
- [Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models](https://mlsanity.com/paper/2405.04233/) (2024)
- [In-Context LoRA for Diffusion Transformers](https://mlsanity.com/paper/2410.23775/) (2024)

## Built on by (most cited first, 27 of 27)

- [DreamX-World 1.0: A General-Purpose Interactive World Model](https://mlsanity.com/paper/2606.16993/) (2026)
- [Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models](https://mlsanity.com/paper/2508.12880/) (2025)
- [Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning](https://mlsanity.com/paper/2512.24146/) (2025)
- [Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models](https://mlsanity.com/paper/2603.22212/) (2026)
- [Video-As-Prompt: Unified Semantic Control for Video Generation](https://mlsanity.com/paper/2510.20888/) (2025)
- [VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learning](https://mlsanity.com/paper/2510.25772/) (2025)
- [TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment](https://mlsanity.com/paper/2601.05729/) (2026)
- [ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraints](https://mlsanity.com/paper/2510.14847/) (2025)
- [From Scale to Speed: Adaptive Test-Time Scaling for Image Editing](https://mlsanity.com/paper/2603.00141/) (2026)
- [OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer](https://mlsanity.com/paper/2601.14250/) (2026)
- [Grounded Forcing: Bridging Time-Independent Semantics and Proximal Dynamics in Autoregressive Video Synthesis](https://mlsanity.com/paper/2604.06939/) (2026)
- [VideoAgent: All-in-One Framework for Video Understanding and Editing](https://mlsanity.com/paper/2606.23327/) (2026)
- [Spanning the Visual Analogy Space with a Weight Basis of LoRAs](https://mlsanity.com/paper/2602.15727/) (2026)
- [Embedding-perturbed Exploration Preference Optimization for Flow Models](https://mlsanity.com/paper/2605.15803/) (2026)
- [IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning](https://mlsanity.com/paper/2512.15635/) (2025)
- [Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation](https://mlsanity.com/paper/2512.24271/) (2025)
- [Over++: Generative Video Compositing for Layer Interaction Effects](https://mlsanity.com/paper/2512.19661/) (2025)
- [Learning Explicit Physical Parameter Control and Benchmarking for Video Generation](https://mlsanity.com/paper/2607.18924/) (2026)
- [EffectMaker: Unifying Reasoning and Generation for Customized Visual Effect Creation](https://mlsanity.com/paper/2603.06014/) (2026)
- [CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal](https://mlsanity.com/paper/2603.21901/) (2026)
- [ConceptWeaver: Weaving Disentangled Concepts with Flow](https://mlsanity.com/paper/2603.28493/) (2026)
- [EasyVFX: Frequency-Driven Decoupling for Resource-Efficient VFX Generation](https://mlsanity.com/paper/2605.22051/) (2026)
- [Enhancing Train-Free Infinite-Frame Generation for Consistent Long Videos](https://mlsanity.com/paper/2605.18233/) (2026)
- [InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generation](https://mlsanity.com/paper/2512.21788/) (2025)
- [Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA](https://mlsanity.com/paper/2603.08210/) (2026)
- [Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generation](https://mlsanity.com/paper/2606.13971/) (2026)
- [EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal](https://mlsanity.com/paper/2608.05565/) (2026)

## Abstract (arXiv)

Visual effects (VFX) are essential visual enhancements fundamental to modern cinematic production. Although video generation models offer cost-efficient solutions for VFX production, current methods are constrained by per-effect LoRA training, which limits generation to single effects. This fundamental limitation impedes applications that require spatially controllable composite effects, i.e., the concurrent generation of multiple effects at designated locations. However, integrating diverse effects into a unified framework faces major challenges: interference from effect variations and spatial uncontrollability during multi-VFX joint training. To tackle these challenges, we propose Omni-Effects, a first unified framework capable of generating prompt-guided effects and spatially controllable composite effects. The core of our framework comprises two key innovations: (1) LoRA-based Mixture of Experts (LoRA-MoE), which employs a group of expert LoRAs, integrating diverse effects within a unified model while effectively mitigating cross-task interference. (2) Spatial-Aware Prompt (SAP) incorporates spatial mask information into the text token, enabling precise spatial control. Furthermore, we introduce an Independent-Information Flow (IIF) module integrated within the SAP, isolating the control signals corresponding to individual effects to prevent any unwanted blending. To facilitate this research, we construct a comprehensive VFX dataset Omni-VFX via a novel data collection pipeline combining image editing and First-Last Frame-to-Video (FLF2V) synthesis, and introduce a dedicated VFX evaluation framework for validating model performance. Extensive experiments demonstrate that Omni-Effects achieves precise spatial control and diverse effect generation, enabling users to specify both the category and location of desired effects.