mlsanityData to 7 Oct 2026

FreeCond: Free Lunch in the Input Conditions of Text-Guided Inpainting

arXiv 2412.00427 ↗First posted 30 Nov 2024cs.CV cs.AIcited by 4

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

In this study, we aim to determine and solve the deficiency of Stable Diffusion Inpainting (SDI) in following the instruction of both prompt and mask. Due to the training bias from masking, the inpainting quality is hindered when the prompt instruction and image condition are not related. Therefore, we conduct a detailed analysis of the internal representations learned by SDI, focusing on how the mask input influences the cross-attention layer. We observe that adapting text key tokens toward the input mask enables the model to selectively paint within the given area. Leveraging these insights, we propose FreeCond, which adjusts only the input mask condition and image condition. By increasing the latent mask value and modifying the frequency of image condition, we align the cross-attention features with the model's training bias to improve generation quality without additional computation, particularly when user inputs are complicated and deviate from the training setup. Extensive experiments demonstrate that FreeCond can enhance any SDI-based model, e.g., yielding up to a 60% and 58% improvement of SDI and SDXLI in the CLIP score.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 59 of its 72 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

59 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion11 Mar
  2. 2023HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models21 Dec
  3. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model11 Oct
  4. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model9 Dec
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2023Segment Anythingcited by 8,158
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
38 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  5. 2017A Closer Look at Memorization in Deep Networkscited by 936
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  8. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  9. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  10. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  13. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  14. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  15. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  16. 2022Blended Latent Diffusioncited by 430
  17. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  18. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  19. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  20. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 243
  21. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 240
  22. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  23. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  24. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  25. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  26. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 194
  27. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  28. 2019Pluralistic Image Completioncited by 162
  29. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  30. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  31. 2021Aggregated Contextual Transformations for High-Resolution Image Inpaintingcited by 89
  32. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  33. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  34. 2024Towards Understanding the Working Mechanism of Text-to-Image Diffusion Modelcited by 48
  35. 2024Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generationcited by 43
  36. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  37. 2024MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editingcited by 30
  38. 2019On the Units of GANs (Extended Abstract)cited by 19

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Modelscited by 6
  2. 2026InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpaintingcited by 2
  3. 2025Token Painter: Training-Free Text-Guided Image Inpainting via Mask Autoregressive Modelscited by 0
  4. 2026Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FreeInpaint: Tuning-free Prompt Alignment and Visual Rationality Enhancement in Image Inpainting
  2. 2024I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting
  3. 2024PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control
  4. 2025IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed
  5. 2025GuidPaint: Class-Guided Image Inpainting with Diffusion Models
  6. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models
  7. 2024Coherent and Multi-modality Image Inpainting via Latent Space Optimization
  8. 2025One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting