mlsanityData to 7 Oct 2026

Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion

arXiv 2405.15313 ↗First posted 24 May 2024cs.CVcited by 1

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recently, text-to-image (T2I) editing has been greatly pushed forward by applying diffusion models. Despite the visual promise of the generated images, inconsistencies with the expected textual prompt remain prevalent. This paper aims to systematically improve the text-guided image editing techniques based on diffusion models, by addressing their limitations. Notably, the common idea in diffusion-based editing firstly reconstructs the source image via inversion techniques e.g., DDIM Inversion. Then following a fusion process that carefully integrates the source intermediate (hidden) states (obtained by inversion) with the ones of the target image. Unfortunately, such a standard pipeline fails in many cases due to the interference of texture retention and the new characters creation in some regions. To mitigate this, we incorporate human annotation as an external knowledge to confine editing within a ``Mask-informed'' region. Then we carefully Fuse the edited image with the source image and a constructed intermediate image within the model's Self-Attention module. Extensive empirical results demonstrate the proposed ``MaSaFusion'' significantly improves the existing T2I editing techniques.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 41 of its 42 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

41 of 42 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing2 Nov
  2. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code2 Oct
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Models8 Dec
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  9. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  10. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attention10 Oct
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
2 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  2. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
15 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  4. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  7. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  10. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  11. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  12. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  13. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 167
  14. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Modelscited by 71
  15. 2023On the Generalization of Diffusion Modelcited by 30

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models
  2. 2024Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks
  3. 2024Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing
  4. 2022Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models
  5. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models
  6. 2023Textual and Visual Prompt Fusion for Image Editing via Step-Wise Alignment
  7. 2023Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
  8. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing