mlsanityData to 7 Oct 2026

Learning from Mistakes: Iterative Prompt Relabeling for Text-to-Image Diffusion Model Training

arXiv 2312.16204 ↗First posted 23 Dec 2023cs.CVcited by 5

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have shown impressive performance in many domains. However, the model's capability to follow natural language instructions (e.g., spatial relationships between objects, generating complex scenes) is still unsatisfactory. In this work, we propose Iterative Prompt Relabeling (IPR), a novel algorithm that aligns images to text through iterative image sampling and prompt relabeling with feedback. IPR first samples a batch of images conditioned on the text, then relabels the text prompts of unmatched text-image pairs with classifier feedback. We conduct thorough experiments on SDv2 and SDXL, testing their capability to follow instructions on spatial relations. With IPR, we improved up to 15.22% (absolute improvement) on the challenging spatial relation VISOR benchmark, demonstrating superior performance compared to previous RL methods. Our code is publicly available at https://github.com/xinyan-cxy/IPR-RLDF.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

31 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Controllable Text-to-Image Generation with GPT-429 May
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
15 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2017Deep Reinforcement Learning: An Overviewcited by 327
  12. 2019Zero-Shot Semantic Segmentationcited by 245
  13. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  14. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  15. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  2. 2026ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Taskscited by 6
  3. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  4. 2025InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Modelscited by 1
  5. 2026PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Reverse Stable Diffusion: What prompt was used to generate this image?
  2. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  3. 2025EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models
  4. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  5. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learning
  6. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  7. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  8. 2024Relation Rectification in Diffusion Model