mlsanityData to 14 Oct 2026

DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback

arXiv 2311.17946 ↗First posted 29 Nov 2023cs.CV cs.AI cs.CLcited by 58

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite their wide-spread success, Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user's input text. We introduce DreamSync, a model-agnostic training algorithm by design that improves T2I models to be faithful to the text input. DreamSync builds off a recent insight from TIFA's evaluation framework -- that large vision-language models (VLMs) can effectively identify the fine-grained discrepancies between generated images and the text inputs. DreamSync uses this insight to train T2I models without any labeled data; it improves T2I models using its own generations. First, it prompts the model to generate several candidate images for a given input text. Then, it uses two VLMs to select the best generation: a Visual Question Answering model that measures the alignment of generated images to the text, and another that measures the generation's aesthetic quality. After selection, we use LoRA to iteratively finetune the T2I model to guide its generation towards the selected best generations. DreamSync does not need any additional human annotation. model architecture changes, or reinforcement learning. Despite its simplicity, DreamSync improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA aesthetic) and human evaluation.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

40 of 56 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 982
28 more
  1. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  5. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  6. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  8. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  9. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  10. 2016Visual Relationship Detection with Language Priorscited by 334
  11. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  12. 2019Connecting Vision and Language with Localized Narrativescited by 226
  13. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 225
  14. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199
  15. 2023Teaching CLIP to Count to Tencited by 198
  16. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 171
  17. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  18. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 154
  19. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  20. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  21. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  22. 2022Character-Aware Models Improve Visual Text Renderingcited by 92
  23. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  24. 2023VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretrainingcited by 65
  25. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 51
  26. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
  27. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  28. 2023SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examplescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 19; 2026: 10.

58 papers.

Sort:

By area:

  1. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  3. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  4. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  5. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  8. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
  9. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 61
  10. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  11. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  12. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
38 more
  1. 2024Reward Guided Latent Consistency Distillationcited by 35
  2. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  3. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 33
  4. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  5. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  6. 2025MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generationcited by 23
  7. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  8. 2025Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformerscited by 20
  9. 2025PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimizationcited by 19
  10. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  11. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  12. 2025Eval3D: Interpretable and Fine-grained Evaluation for 3D Generationcited by 16
  13. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  14. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  15. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  16. 2025Science-T2I: Addressing Scientific Illusions in Image Synthesiscited by 15
  17. 2024SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Datacited by 14
  18. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 10
  19. 2024AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generationcited by 9
  20. 2024EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Modelscited by 9
  21. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 9
  22. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  23. 2025ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answeringcited by 9
  24. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 7
  25. 2024Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Modelscited by 5
  26. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  27. 2025CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedbackcited by 4
  28. 2024Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Trainingcited by 3
  29. 2024GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Designcited by 2
  30. 2025CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Artcited by 2
  31. 2025Bounding Box-Guided Diffusion for Synthesizing Industrial Images and Segmentation Mapcited by 2
  32. 2026Personalized Image Generation via Human-in-the-loop Bayesian Optimizationcited by 2
  33. 2026Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklistcited by 2
  34. 2025CoherenDream: Boosting Holistic Text Coherence in 3D Generation via Multimodal Large Language Models Feedbackcited by 1
  35. 2025Structured Information for Improving Spatial Relationships in Text-to-Image Generationcited by 1
  36. 2025FoR-SALE: Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editingcited by 1
  37. 2025Learning by Analogy: A Causal Framework for Composition Generalizationcited by 1
  38. 2024ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Imagescited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization
  2. 2025Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Models
  3. 2025Test-time Prompt Refinement for Text-to-Image Models
  4. 2024Bridging Different Language Models and Generative Vision Models for Text-to-Image Generation
  5. 2026Iterative Refinement Improves Compositional Image Generation
  6. 2025Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment
  7. 2025M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
  8. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark