mlsanityData to 7 Oct 2026

Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts

arXiv 2305.08850 ↗First posted 15 May 2023cs.CVcited by 33

On the map: Diffusion models → Video generation

Abstract (arXiv)

The text-driven image and video diffusion models have achieved unprecedented success in generating realistic and diverse content. Recently, the editing and variation of existing images and videos in diffusion-based generative models have garnered significant attention. However, previous works are limited to editing content with text or providing coarse personalization using a single visual clue, rendering them unsuitable for indescribable content that requires fine-grained and detailed control. In this regard, we propose a generic video editing framework called Make-A-Protagonist, which utilizes textual and visual clues to edit videos with the goal of empowering individuals to become the protagonists. Specifically, we leverage multiple experts to parse source video, target visual and textual clues, and propose a visual-textual-based video generation model that employs mask-guided denoising sampling to generate the desired output. Extensive results demonstrate the versatile and remarkable editing capabilities of Make-A-Protagonist.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 42 of its 46 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

42 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  2. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  3. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  7. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2023Segment Anythingcited by 8,158
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
22 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  13. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  14. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  15. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  16. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  17. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  18. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  19. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  20. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  21. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  22. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 13; 2025: 7; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  3. 2023A Survey on Video Diffusion Modelscited by 203
  4. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  5. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  6. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  7. 2024MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modelingcited by 63
  8. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  9. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54
  10. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  11. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  12. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
21 more
  1. 2023ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systemscited by 28
  2. 2024Video Diffusion Models: A Surveycited by 28
  3. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  4. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  5. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  6. 2025OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Modelscited by 9
  7. 2024GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Modelscited by 7
  8. 2023DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editingcited by 6
  9. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6
  10. 2026Evolution of Video Generative Foundationscited by 6
  11. 2024AudioScenic: Audio-Driven Video Scene Editingcited by 5
  12. 2024Segment Anything for Videos: A Systematic Surveycited by 5
  13. 2024Motion by Queries: Identity-Motion Trade-offs in Text-to-Video Generationcited by 3
  14. 2023RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editingcited by 2
  15. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  16. 2025AnyCharV: Bootstrap Controllable Character Video Generation with Fine-to-Coarse Guidancecited by 2
  17. 2025RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editingcited by 1
  18. 2025A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentationcited by 1
  19. 2023SmoothVideo: Smooth Video Synthesis with Noise Constraints on Diffusion Models for One-shot Video Tuningcited by 0
  20. 2025Good Noise Makes Good Edits: A Training-Free Diffusion-Based Video Editing with Image and Text Promptscited by 0
  21. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Models
  2. 2024MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
  3. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising
  4. 2023Pix2Video: Video Editing using Image Diffusion
  5. 2026Vera: A Layered Diffusion Model for Content-Preserving Video Editing
  6. 2023MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
  7. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  8. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editing