# Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement

arXiv: https://arxiv.org/abs/2406.16272 · HTML: https://mlsanity.com/paper/2406.16272/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [Attribute2Image: Conditional Image Generation from Visual Attributes](https://mlsanity.com/paper/1512.00570/) (2015)
- [Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning](https://mlsanity.com/paper/2305.04175/) (2023)
- [Understanding Hallucinations in Diffusion Models through Mode Interpolation](https://mlsanity.com/paper/2406.09358/) (2024)
- [Generating images of rare concepts using pre-trained diffusion models](https://mlsanity.com/paper/2304.14530/) (2023)
- [Improving Text-to-Image Synthesis Using Contrastive Learning](https://mlsanity.com/paper/2107.02423/) (2021)
- [Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy](https://mlsanity.com/paper/2405.14800/) (2024)
- [Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakes](https://mlsanity.com/paper/2304.06470/) (2023)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search](https://mlsanity.com/paper/2306.00974/) (2023)
- [Groot: Adversarial Testing for Generative Text-to-Image Models with Tree-based Semantic Transformation](https://mlsanity.com/paper/2402.12100/) (2024)

## Built on by (most cited first, 5 of 5)

- [Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models](https://mlsanity.com/paper/2504.01137/) (2025)
- [Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.25994/) (2026)
- [Mapping the Mind of an Instruction-based Image Editing using SMILE](https://mlsanity.com/paper/2412.16277/) (2024)
- [Taming the Tri-Space Tension: ARC-Guided Hallucination Modeling and Control for Text-to-Image Generation](https://mlsanity.com/paper/2507.04946/) (2025)
- [Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time](https://mlsanity.com/paper/2607.26647/) (2026)

## Abstract (arXiv)

Text-to-Image Diffusion Models (T2I DMs) have garnered significant attention for their ability to generate high-quality images from textual descriptions. However, these models often produce images that do not fully align with the input prompts, resulting in semantic inconsistencies. The most prominent issue among these semantic inconsistencies is catastrophic-neglect, where the images generated by T2I DMs miss key objects mentioned in the prompt. We first conduct an empirical study on this issue, exploring the prevalence of catastrophic-neglect, potential mitigation strategies with feature enhancement, and the insights gained. Guided by the empirical findings, we propose an automated repair approach named Patcher to address catastrophic-neglect in T2I DMs. Specifically, Patcher first determines whether there are any neglected objects in the prompt, and then applies attention-guided feature enhancement to these neglected objects, resulting in a repaired prompt. Experimental results on three versions of Stable Diffusion demonstrate that Patcher effectively repairs the issue of catastrophic-neglect, achieving 10.1%-16.3% higher Correct Rate in image generation compared to baselines.