# A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation

arXiv: https://arxiv.org/abs/2310.16656 · HTML: https://mlsanity.com/paper/2310.16656/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment](https://mlsanity.com/paper/2306.08877/) (2023)
- [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023)
- [Improving Multimodal Datasets with Image Captioning](https://mlsanity.com/paper/2307.10350/) (2023)
- [Semantic Object Accuracy for Generative Text-to-Image Synthesis](https://mlsanity.com/paper/1910.13321/) (2019)
- [Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2304.03869/) (2023)
- [Image Captioning with Multi-Context Synthetic Data](https://mlsanity.com/paper/2305.18072/) (2023)
- [The Role of Data Curation in Image Captioning](https://mlsanity.com/paper/2305.03610/) (2023)

## Built on by (most cited first, 30 of 41)

- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization](https://mlsanity.com/paper/2404.04650/) (2024)
- [Aligning Diffusion Models by Optimizing Human Utility](https://mlsanity.com/paper/2404.04465/) (2024)
- [DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing](https://mlsanity.com/paper/2402.02583/) (2024)
- [Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2402.10210/) (2024)
- [From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuning](https://mlsanity.com/paper/2504.16080/) (2025)
- [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024)
- [VideoTetris: Towards Compositional Text-to-Video Generation](https://mlsanity.com/paper/2406.04277/) (2024)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [TokenCompose: Text-to-Image Diffusion with Token-level Supervision](https://mlsanity.com/paper/2312.03626/) (2023)
- [PALP: Prompt Aligned Personalization of Text-to-Image Models](https://mlsanity.com/paper/2401.06105/) (2024)
- [Reward Guided Latent Consistency Distillation](https://mlsanity.com/paper/2403.11027/) (2024)
- [All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds](https://mlsanity.com/paper/2411.18810/) (2024)
- [Getting it Right: Improving Spatial Consistency in Text-to-Image Models](https://mlsanity.com/paper/2404.01197/) (2024)
- [Personalized Preference Fine-tuning of Diffusion Models](https://mlsanity.com/paper/2501.06655/) (2025)
- [SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data](https://mlsanity.com/paper/2403.06952/) (2024)
- [Don't drop your samples! Coherence-aware training benefits Conditional diffusion](https://mlsanity.com/paper/2405.20324/) (2024)
- [ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation](https://mlsanity.com/paper/2410.01731/) (2024)
- [AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation](https://mlsanity.com/paper/2403.13352/) (2024)
- [Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.16503/) (2024)
- [Video-GPT via Next Clip Diffusion](https://mlsanity.com/paper/2505.12489/) (2025)
- [CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2412.13195/) (2024)
- [ReCorD: Reasoning and Correcting Diffusion for HOI Generation](https://mlsanity.com/paper/2407.17911/) (2024)
- [Pixel-Space Post-Training of Latent Diffusion Models](https://mlsanity.com/paper/2409.17565/) (2024)
- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning](https://mlsanity.com/paper/2507.22604/) (2025)
- [EmoCtrl: Controllable Emotional Image Content Generation](https://mlsanity.com/paper/2512.22437/) (2025)
- [Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition](https://mlsanity.com/paper/2402.15504/) (2024)
- [Improving Text Generation on Images with Synthetic Captions](https://mlsanity.com/paper/2406.00505/) (2024)
- [PC2: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval](https://mlsanity.com/paper/2408.01349/) (2024)

## Abstract (arXiv)

Text-to-image diffusion models achieved a remarkable leap in capabilities over the last few years, enabling high-quality and diverse synthesis of images from a textual prompt. However, even the most advanced models often struggle to precisely follow all of the directions in their prompts. The vast majority of these models are trained on datasets consisting of (image, caption) pairs where the images often come from the web, and the captions are their HTML alternate text. A notable example is the LAION dataset, used by Stable Diffusion and other models. In this work we observe that these captions are often of low quality, and argue that this significantly affects the model's capability to understand nuanced semantics in the textual prompts. We show that by relabeling the corpus with a specialized automatic captioning model and training a text-to-image model on the recaptioned dataset, the model benefits substantially across the board. First, in overall image quality: e.g. FID 14.84 vs. the baseline of 17.87, and 64.3% improvement in faithful image generation according to human evaluation. Second, in semantic alignment, e.g. semantic object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the corpus and provide evidence that this technique, which we call RECAP, both reduces the train-inference discrepancy and provides the model with more information per example, increasing sample efficiency and allowing the model to better understand the relations between captions and images.