mlsanityData to 14 Oct 2026

CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility

arXiv 2403.12035 ↗First posted 18 Mar 2024cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Recent advancements in video generation have been remarkable, yet many existing methods struggle with issues of consistency and poor text-video alignment. Moreover, the field lacks effective techniques for text-guided video inpainting, a stark contrast to the well-explored domain of text-guided image inpainting. To this end, this paper proposes a novel text-guided video inpainting model that achieves better consistency, controllability and compatibility. Specifically, we introduce a simple but efficient motion capture module to preserve motion consistency, and design an instance-aware region selection instead of a random region selection to obtain better textual controllability, and utilize a novel strategy to inject some personalized models into our CoCoCo model and thus obtain better model compatibility. Extensive experiments show that our model can generate high-quality video clips. Meanwhile, our model shows better motion consistency, textual controllability and model compatibility. More details are shown in [cococozibojia.github.io](cococozibojia.github.io).

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 49 of its 59 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

49 of 59 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2023Segment Anythingcited by 8,234
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
37 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  9. 2022Editing Models with Task Arithmeticcited by 1,120
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  12. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  13. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  14. 2023ModelScope Text-to-Video Technical Reportcited by 658
  15. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  16. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  17. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  18. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  19. 2023CoTracker: It is Better to Track Togethercited by 522
  20. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  21. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  22. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  23. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  24. 2022Blended Latent Diffusioncited by 432
  25. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  26. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  27. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  28. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  29. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  30. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  31. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 126
  32. 2021Paint by Wordcited by 107
  33. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 82
  34. 2024Magic-Me: Identity-Specific Video Customized Diffusioncited by 59
  35. 2022Diffusion Guided Domain Adaptation of Image Generatorscited by 35
  36. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Modelscited by 31
  37. 2023Hierarchical Masked 3D Diffusion Model for Video Outpaintingcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 14.

50 papers.

Sort:

By area:

  1. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 84
  2. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  3. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  4. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 45
  5. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  6. 2024Generative Omnimatte: Learning to Decompose Video into Layerscited by 38
  7. 2025ROSE: Remove Objects with Side Effects in Videoscited by 38
  8. 2024Generative Video Propagationcited by 36
  9. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  10. 2024StableV2V: Stablizing Shape Consistency in Video-to-Video Editingcited by 26
  11. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 25
  12. 2024Video Diffusion Models are Strong Video Inpaintercited by 21
38 more
  1. 2025OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulationcited by 19
  2. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  3. 2026EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasingcited by 15
  4. 2025MTV-Inpaint: Multi-Task Long Video Inpaintingcited by 14
  5. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  6. 2024UniPaint: Unified Space-time Video Inpainting via Mixture-of-Expertscited by 13
  7. 2025See4D: Pose-Free 4D Generation via Auto-Regressive Video Inpaintingcited by 12
  8. 2025Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpaintingcited by 11
  9. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 10
  10. 2025Consistent Video Editing as Flow-Driven Image-to-Video Generationcited by 8
  11. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  12. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 7
  13. 2024Coherent Video Inpainting Using Optical Flow-Guided Efficient Diffusioncited by 6
  14. 2025iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformercited by 5
  15. 2025LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localizationcited by 5
  16. 2026From Ideal to Real: Stable Video Object Removal under Imperfect Conditionscited by 5
  17. 2025CoCo4D: Comprehensive and Complex 4D Scene Generationcited by 4
  18. 2025DFVEdit: Conditional Delta Flow Vector for Zero-shot Video Editingcited by 4
  19. 2025OutDreamer: Video Outpainting with a Diffusion Transformercited by 4
  20. 2026NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editingcited by 4
  21. 2026LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attentioncited by 4
  22. 2025Enabling Versatile Controls for Video Diffusion Modelscited by 3
  23. 2025FakeParts: a New Family of AI-Generated DeepFakescited by 3
  24. 2026From Understanding to Erasing: Towards Complete and Stable Video Object Removalcited by 3
  25. 2026Physics-Aware Video Instance Removal Benchmarkcited by 3
  26. 2025BulletGen: Improving 4D Reconstruction with Bullet-Time Generationcited by 2
  27. 2025VIP: Video Inpainting Pipeline for Real World Human Removalcited by 1
  28. 2025Refaccade: Editing Object with Given Reference Texturecited by 1
  29. 2026BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removalcited by 1
  30. 2024Track Anything Behind Everything: Zero-Shot Amodal Video Object Segmentationcited by 0
  31. 2025DreamDance: Animating Character Art via Inpainting Stable Gaussian Worldscited by 0
  32. 2025Enhancing Video Inpainting with Aligned Frame Interval Guidancecited by 0
  33. 2026Controllable Video Object Insertion via Multi-View Priorscited by 0
  34. 2026AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidancecited by 0
  35. 2026Vera: A Layered Diffusion Model for Content-Preserving Video Editingcited by 0
  36. 2026ScaleVid: Geometry-Aware Video Object Scaling with Mesh-Free Inferencecited by 0
  37. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0
  38. 2026StreamErase: Towards Ultra-Fast Streaming Video Erasingcited by 0