mlsanityData to 7 Oct 2026

A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting

arXiv 2312.03594 ↗First posted 6 Dec 2023cs.CVcited by 168

Not on the map. Why?

Cited by 61 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Advancing image inpainting is challenging as it requires filling user-specified regions for various intents, such as background filling and object synthesis. Existing approaches focus on either context-aware filling or object synthesis using text descriptions. However, achieving both tasks simultaneously is challenging due to differing training strategies. To overcome this challenge, we introduce PowerPaint, the first high-quality and versatile inpainting model that excels in multiple inpainting tasks. First, we introduce learnable task prompts along with tailored fine-tuning strategies to guide the model's focus on different inpainting targets explicitly. This enables PowerPaint to accomplish various inpainting tasks by utilizing different task prompts, resulting in state-of-the-art performance. Second, we demonstrate the versatility of the task prompt in PowerPaint by showcasing its effectiveness as a negative prompt for object removal. Moreover, we leverage prompt interpolation techniques to enable controllable shape-guided object inpainting, enhancing the model's applicability in shape-guided applications. Finally, we conduct extensive experiments and applications to verify the effectiveness of PowerPaint. We release our codes and models on our project page: https://powerpaint.github.io/.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

34 of 46 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
22 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  9. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  10. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  11. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  12. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  13. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  14. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 161
  15. 2019Learning Pyramid-Context Encoder Network for High-Quality Image Inpaintingcited by 94
  16. 2021Aggregated Contextual Transformations for High-Resolution Image Inpaintingcited by 89
  17. 2017Generative Face Completioncited by 79
  18. 2019Boundless: Generative Adversarial Networks for Image Extensioncited by 62
  19. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Modelcited by 60
  20. 2021In&Out : Diverse Image Outpainting via GAN Inversioncited by 55
  21. 2019Very Long Natural Scenery Image Prediction by Outpaintingcited by 50
  22. 2023LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Modelcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 37; 2025: 85; 2026: 44.

168 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  2. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  5. 2024Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advancescited by 105
  6. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  7. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  8. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  9. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 47
  10. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 42
  11. 2025OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpaintingcited by 38
  12. 2025LBM: Latent Bridge Matching for Fast Image-to-Image Translationcited by 36
38 more
  1. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
  2. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  3. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  4. 2024BrushEdit: All-In-One Image Inpainting and Editingcited by 31
  5. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
  6. 2023PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Modelscited by 30
  7. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  8. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  9. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 30
  10. 2025DragFlow: Unleashing DiT Priors with Region Based Supervision for Drag Editingcited by 29
  11. 2024DepthLab: From Partial to Completecited by 27
  12. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  13. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 24
  14. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  15. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  16. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  17. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  18. 2023Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistencycited by 19
  19. 2024RAD: Region-Aware Diffusion Models for Image Inpaintingcited by 19
  20. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Modelscited by 18
  21. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  22. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  23. 2024InsightEdit: Towards Better Instruction Following for Image Editingcited by 17
  24. 2025Erase Diffusion: Empowering Object Removal Through Calibrating Diffusion Pathwayscited by 17
  25. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  26. 2024Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflectionscited by 16
  27. 2025Zooming In on Fakes: A Novel Dataset for Localized AI-Generated Image Detection with Forgery Amplification Approachcited by 16
  28. 2024MagicEraser: Erasing Any Objects via Semantics-Aware Controlcited by 15
  29. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  30. 2025MTV-Inpaint: Multi-Task Long Video Inpaintingcited by 14
  31. 2025CompBench: Benchmarking Complex Instruction-guided Image Editingcited by 14
  32. 2024UniPaint: Unified Space-time Video Inpainting via Mixture-of-Expertscited by 13
  33. 2025Precise Object and Effect Removal with Adaptive Target-Aware Attentioncited by 13
  34. 2025OmniTry: Virtual Try-On Anything without Maskscited by 13
  35. 2025LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Samplingcited by 11
  36. 2025Visual Jenga: Discovering Object Dependencies via Counterfactual Inpaintingcited by 11
  37. 2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
  38. 2025OmniVTON: Training-Free Universal Virtual Try-Oncited by 9

50 most cited of 168 All 168 (JSON)