Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
Not on the map. Why?
Abstract (arXiv)
Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7% for components with two capabilities to 37.2% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE's 89.51-point estimate with 21.3% less GPU-s/MP.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 25 Aug 2026.
Builds on
21 of 26 references.
References · 21
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
- 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
- 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
- 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 277
- 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
- 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
- 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
- 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 87
- 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 69
9 more
- 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 61
- 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 39
- 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 38
- 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
- 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 32
- 2026Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluationcited by 22
- 2026OctoT2I: A Self-Evolving Agentic Text-to-Image Routercited by 5
- 2025Cost-Aware Routing for Efficient Text-To-Image Generationcited by 4
- 2025Image-POSER: Reflective RL for Multi-Expert Image Generation and Editingcited by 2
Built on by
Not cited yet.