# Text-to-Image Generation Grounded by Fine-Grained User Attention

arXiv: https://arxiv.org/abs/2011.03775 · HTML: https://mlsanity.com/paper/2011.03775/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2020.

## Built on by (most cited first, 30 of 36)

- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention](https://mlsanity.com/paper/2403.11052/) (2024)
- [Text to Image Generation with Semantic-Spatial Aware GAN](https://mlsanity.com/paper/2104.00567/) (2021)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization](https://mlsanity.com/paper/2110.10834/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2203.15799/) (2022)
- [Simple and Effective Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2204.02960/) (2022)
- [ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations](https://mlsanity.com/paper/2312.04655/) (2023)
- [CRD-CGAN: Category-Consistent and Relativistic Constraints for Diverse Text-to-Image Generation](https://mlsanity.com/paper/2107.13516/) (2021)
- [Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides](https://mlsanity.com/paper/2208.08080/) (2022)
- [ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.04695/) (2023)
- [NarrativePlay: Interactive Narrative Understanding](https://mlsanity.com/paper/2310.01459/) (2023)
- [Markup-to-Image Diffusion Models with Scheduled Sampling](https://mlsanity.com/paper/2210.05147/) (2022)
- [IC3D: Image-Conditioned 3D Diffusion for Shape Generation](https://mlsanity.com/paper/2211.10865/) (2022)
- [DT2I: Dense Text-to-Image Generation from Region Descriptions](https://mlsanity.com/paper/2204.02035/) (2022)
- [Text-to-Image Generation via Implicit Visual Guidance and Hypernetwork](https://mlsanity.com/paper/2208.08493/) (2022)
- [MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives](https://mlsanity.com/paper/2501.04184/) (2025)
- [The Right Losses for the Right Gains: Improving the Semantic Consistency of Deep Text-to-Image Generation with Distribution-Sensitive Losses](https://mlsanity.com/paper/2312.10854/) (2023)
- [Emage: Non-Autoregressive Text-to-Image Generation](https://mlsanity.com/paper/2312.14988/) (2023)
- [Offline Evaluation of Set-Based Text-to-Image Generation](https://mlsanity.com/paper/2410.17331/) (2024)

## Abstract (arXiv)

Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.