mlsanityData to 14 Oct 2026

Towards Language-Driven Video Inpainting via Multimodal Large Language Models

arXiv 2401.10226 ↗First posted 18 Jan 2024cs.CVcited by 33

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We will make datasets, code, and models publicly available.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 72 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

61 of 72 references.

References · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2014Deep Learning Face Attributes in the Wildcited by 3,693
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  13. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  14. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  15. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  16. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  17. 2018Video-to-Video Synthesiscited by 390
  18. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  19. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 330
  20. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 278
  21. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  22. 2018Learning Blind Video Temporal Consistencycited by 253
  23. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  24. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 232
  25. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  26. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 218
  27. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  28. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  29. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 163
  30. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  31. 2020Semantic Flow for Fast and Accurate Scene Parsingcited by 128
  32. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  33. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 113
  34. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  35. 2020Recurrent Feature Reasoning for Image Inpaintingcited by 107
  36. 2018Actor and Action Video Segmentation from a Sentencecited by 106
  37. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  38. 2021FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpaintingcited by 95

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 9.

33 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  3. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
  4. 2025DiffuEraser: A Diffusion Model for Video Inpaintingcited by 72
  5. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  6. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  7. 2024Generative Omnimatte: Learning to Decompose Video into Layerscited by 38
  8. 2025ROSE: Remove Objects with Side Effects in Videoscited by 38
  9. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 18
  10. 2025DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistencycited by 15
  11. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  12. 2025Guiding Audio Editing with Audio Language Modelcited by 12
21 more
  1. 2025TACO: Taming Diffusion for in-the-wild Video Amodal Completioncited by 10
  2. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 9
  3. 2024ActAnywhere: Subject-Aware Video Background Generationcited by 8
  4. 2024MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusioncited by 7
  5. 2025LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localizationcited by 5
  6. 2025BVINet: Unlocking Blind Video Inpainting with Zero Annotationscited by 4
  7. 2025Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removalcited by 4
  8. 2025FakeParts: a New Family of AI-Generated DeepFakescited by 3
  9. 2026From Understanding to Erasing: Towards Complete and Stable Video Object Removalcited by 3
  10. 2024MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learningcited by 2
  11. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generationcited by 2
  12. 2026Diff2DGS: Reliable Reconstruction of Occluded Surgical Scenes via 2D Gaussian Splattingcited by 2
  13. 2026SPLIT: Training-Free AI-Generated and Partially Edited Video Detection via Spatial Patch-Level Incoherence and Temporal Roughnesscited by 2
  14. 2026ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagationcited by 1
  15. 2026Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Modelcited by 1
  16. 2026From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Plantingcited by 1
  17. 2025Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Modelscited by 0
  18. 2025VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Modelscited by 0
  19. 2026Towards Customized Multimodal Role-Playcited by 0
  20. 2026VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removalcited by 0
  21. 2026AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantically Consistent Video Inpainting with Conditional Diffusion Models
  2. 2025MTV-Inpaint: Multi-Task Long Video Inpainting
  3. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them First
  4. 2025MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
  5. 2024InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Models
  6. 2024Image Inpainting Models are Effective Tools for Instruction-guided Image Editing
  7. 2023AVID: Any-Length Video Inpainting with Diffusion Model
  8. 2024I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting