mlsanityData to 7 Oct 2026

Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation

arXiv 2310.18235 ↗First posted 27 Oct 2023cs.CV cs.AI cs.CLcited by 159

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating text-to-image models is notoriously difficult. A strong recent approach for assessing text-image faithfulness is based on QG/A (question generation and answering), which uses pre-trained foundational models to automatically generate a set of questions and answers from the prompt, and output images are scored based on whether these answers extracted with a visual question answering model are consistent with the prompt-based answers. This kind of evaluation is naturally dependent on the quality of the underlying QG and VQA models. We identify and address several reliability challenges in existing QG/A work: (a) QG questions should respect the prompt (avoiding hallucinations, duplications, and omissions) and (b) VQA answers should be consistent (not asserting that there is no motorcycle in an image while also claiming the motorcycle is blue). We address these issues with Davidsonian Scene Graph (DSG), an empirically grounded evaluation framework inspired by formal semantics, which is adaptable to any QG/A frameworks. DSG produces atomic and unique questions organized in dependency graphs, which (i) ensure appropriate semantic coverage and (ii) sidestep inconsistent answers. With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above. Finally, we present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts, covering a wide range of fine-grained semantic categories with a balanced distribution. We release the DSG-1k prompts and the corresponding DSG questions.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

32 of 43 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
20 more
  1. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  2. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  3. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  4. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  5. 2016Visual Relationship Detection with Language Priorscited by 333
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  7. 2019Connecting Vision and Language with Localized Narrativescited by 225
  8. 2023Teaching CLIP to Count to Tencited by 194
  9. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  10. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  11. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  12. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  13. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  14. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  15. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  16. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  17. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  18. 2022All You May Need for VQA are Image Captionscited by 65
  19. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  20. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 48; 2025: 62; 2026: 43.

159 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  4. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  5. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  6. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  7. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  8. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  9. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  10. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  11. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  12. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
38 more
  1. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  2. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  3. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  4. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 60
  5. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  6. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  7. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  8. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  9. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  10. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 42
  11. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 37
  12. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  13. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  14. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  15. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 33
  16. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  17. 2024CosmicMan: A Text-to-Image Foundation Model for Humanscited by 30
  18. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  19. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  20. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  21. 2024STIV: Scalable Text and Image Conditioned Video Generationcited by 24
  22. 2024CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesiscited by 23
  23. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 23
  24. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
  25. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 21
  26. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Modelscited by 21
  27. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 21
  28. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  29. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
  30. 2025UniVG: A Generalist Diffusion Model for Unified Image Generation and Editingcited by 19
  31. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  32. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  33. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  34. 2024Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)cited by 17
  35. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  36. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 16
  37. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 16
  38. 2026Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluationcited by 16

50 most cited of 159 All 159 (JSON)