mlsanityData to 14 Oct 2026

VACE: All-in-One Video Creation and Editing

arXiv 2503.07598 ↗First posted 10 Mar 2025cs.CVcited by 479

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion Transformer has demonstrated powerful capability and scalability in generating high-quality images and videos. Further pursuing the unification of generation and editing tasks has yielded significant progress in the domain of image content creation. However, due to the intrinsic demands for consistency across both temporal and spatial dynamics, achieving a unified approach for video synthesis remains challenging. We introduce VACE, which enables users to perform Video tasks within an All-in-one framework for Creation and Editing. These tasks include reference-to-video generation, video-to-video editing, and masked video-to-video editing. Specifically, we effectively integrate the requirements of various tasks by organizing video task inputs, such as editing, reference, and masking, into a unified interface referred to as the Video Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we inject different task concepts into the model using formalized representations of temporal and spatial dimensions, allowing it to handle arbitrary video synthesis tasks flexibly. Extensive experiments demonstrate that the unified model of VACE achieves performance on par with task-specific models across various subtasks. Simultaneously, it enables diverse applications through versatile task combinations. Project page: https://ali-vilab.github.io/VACE-Page/.

Key facts. Cited by 479 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 65 of its 74 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

65 of 74 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Goku: Flow Based Video Generative Foundation Models7 Feb
  2. 2025MagicMirror: ID-Preserved Video Generation in Video Diffusion Transformers7 Jan
  3. 2024LTX-Video: Realtime Video Latent Diffusion30 Dec
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  5. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models27 Dec
  6. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  7. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  8. 2023AnyDoor: Zero-shot Object-level Image Customization18 Jul
  9. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning23 May
  10. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar

Other references · 54

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
  1. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  6. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  7. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  8. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  9. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  12. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 436
  13. 2024OmniGen: Unified Image Generationcited by 394
  14. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  15. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  16. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  17. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  18. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  19. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  20. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  21. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  22. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 219
  23. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  24. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  25. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  26. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  27. 2024PuLID: Pure and Lightning ID Customization via Contrastive Alignmentcited by 144
  28. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 129
  29. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 102
  30. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  31. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  32. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  33. 2022Learning to generate line drawings that convey geometry and semanticscited by 83
  34. 2023MagicEdit: High-Fidelity and Temporally Coherent Video Editingcited by 76
  35. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  36. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  37. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  38. 2024Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generationcited by 42

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 148; 2026: 331.

479 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2025Controllable Video Generation: A Surveycited by 98
  3. 2025HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationcited by 87
  4. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  5. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 80
  6. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 79
  7. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  8. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  9. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
  10. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 53
  11. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 46
  12. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 45
38 more
  1. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 40
  2. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 36
  3. 2026Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidancecited by 36
  4. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 35
  5. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  6. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  7. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  8. 2025RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wildcited by 29
  9. 2026ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignmentcited by 29
  10. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  11. 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 28
  12. 2025Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generationcited by 28
  13. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 28
  14. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  15. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Modelscited by 28
  16. 2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
  17. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
  18. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 27
  19. 2026VINO: A Unified Visual Generator with Interleaved OmniModal Contextcited by 26
  20. 2025Skyra: AI-Generated Video Detection via Grounded Artifact Reasoningcited by 25
  21. 2025MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglementcited by 23
  22. 2025BindWeave: Subject-Consistent Video Generation via Cross-Modal Integrationcited by 23
  23. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  24. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 23
  25. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  26. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 22
  27. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 20
  28. 2025OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulationcited by 19
  29. 2025UltraGen: High-Resolution Video Generation with Hierarchical Attentioncited by 18
  30. 2025Generative Neural Video Compression via Video Diffusion Priorcited by 18
  31. 2025Phantom-Data : Towards a General Subject-Consistent Video Generation Datasetcited by 17
  32. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 17
  33. 2025DreamVVT: Mastering Realistic Video Virtual Try-On in the Wild via a Stage-Wise Diffusion Transformer Frameworkcited by 17
  34. 2026Latent Spatial Memory for Video World Modelscited by 17
  35. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 16
  36. 2025Generative Pre-trained Autoregressive Diffusion Transformercited by 16
  37. 2025PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancementcited by 16
  38. 2025Kaleido: Open-Sourced Multi-Subject Reference Video Generation Modelcited by 16

50 most cited of 479 All 479 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025V2Edit: Versatile Video Diffusion Editor for Videos and 3D Scenes
  2. 2025UNIC: Unified In-Context Video Editing
  3. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling
  4. 2026LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows
  5. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  6. 2025DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models
  7. 2024MVOC: a training-free multiple video object composition method with diffusion models
  8. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation