mlsanityData to 14 Oct 2026

AutoVFX: Physically Realistic Video Editing from Natural Language Instructions

arXiv 2411.02394 ↗First posted 4 Nov 2024cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction alignment, editing versatility, and physical plausibility.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 91 of its 114 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

91 of 114 references.

References · 91

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  4. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  12. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  13. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 616
  14. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  15. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  16. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  17. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  18. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  19. 2023Learning Interactive Real-World Simulatorscited by 458
  20. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  21. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  22. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  23. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  24. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
  25. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  26. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  27. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  28. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  29. 2023GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfacescited by 252
  30. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 221
  31. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 217
  32. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  33. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  34. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 179
  35. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  36. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 168
  37. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  38. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 8.

27 papers.

Sort:

By area:

  1. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  2. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  3. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  4. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  5. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  6. 2025Generative Physical AI in Vision: A Surveycited by 24
  7. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 24
  8. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 22
  9. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 14
  10. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14
  11. 2025Controllable Weather Synthesis and Removal with Video Diffusion Modelscited by 11
  12. 2025DynVFX: Augmenting Real Videos with Dynamic Contentcited by 9
15 more
  1. 2025Robot Learning from Any Imagescited by 7
  2. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  3. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 5
  4. 2025PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animationcited by 4
  5. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 4
  6. 2026Versatile Editing of Video Content, Actions, and Dynamics without Trainingcited by 4
  7. 2025LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agentscited by 3
  8. 2025PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenescited by 3
  9. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  10. 2025GOBench: Benchmarking Geometric Optics Generation and Understanding of MLLMscited by 2
  11. 2026FieryGS: In-the-Wild Fire Synthesis with Physics-Integrated Gaussian Splattingcited by 2
  12. 2026OptiWorld: Optimal Control for Video World Generation under Physical Constraintscited by 2
  13. 2026SiPhy: Single-Image Physical Property Reasoningcited by 2
  14. 2026MeGAS: Thermomechanical Dynamic Gaussian Splatting for Thermophysical Scene Editingcited by 0
  15. 2026Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioningcited by 0