mlsanityData to 7 Oct 2026

A-Bench: Are LMMs Masters at Evaluating AI-generated Images?

arXiv 2406.03070 ↗First posted 5 Jun 2024cs.CV cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models. Given the high costs and extensive time commitments required for user studies, many researchers have turned towards employing large multi-modal models (LMMs) as AIGI evaluators, the precision and validity of which are still questionable. Furthermore, traditional benchmarks often utilize mostly natural-captured content rather than AIGIs to test the abilities of LMMs, leading to a noticeable gap for AIGIs. Therefore, we introduce A-Bench in this paper, a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs. Specifically, A-Bench is organized under two key principles: 1) Emphasizing both high-level semantic understanding and low-level visual quality perception to address the intricate demands of AIGIs. 2) Various generative models are utilized for AIGI creation, and various LMMs are employed for evaluation, which ensures a comprehensive validation scope. Ultimately, 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs. We hope that A-Bench will significantly enhance the evaluation process and promote the generation quality for AIGIs. The benchmark is available at https://github.com/Q-Future/A-Bench.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

38 of 55 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
26 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  4. 2016Generative Adversarial Text to Image Synthesiscited by 735
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  12. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  15. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  16. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  17. 2019From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Qualitycited by 221
  18. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  19. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  20. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  21. 2015Generating Images from Captions with Attentioncited by 135
  22. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  23. 2024Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Losscited by 51
  24. 2024AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessmentcited by 33
  25. 2023Exploring the Naturalness of AI-Generated Imagescited by 22
  26. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 21; 2026: 6.

39 papers.

Sort:

By area:

  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  2. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  3. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?cited by 29
  4. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  5. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  6. 2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
  7. 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?cited by 22
  8. 2025Generative Uncertainty in Diffusion Modelscited by 20
  9. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  10. 2025Redundancy Principles for MLLMs Benchmarkscited by 18
  11. 2024HEIE: MLLM-Based Hierarchical Explainable AIGC Image Implausibility Evaluatorcited by 12
  12. 2024MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysiscited by 10
27 more
  1. 2025Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMscited by 9
  2. 2025Information Density Principle for MLLM Benchmarkscited by 8
  3. 2025MoEdit: On Learning Quantity Perception for Multi-object Image Editingcited by 7
  4. 2026Agentic Retoucher for Text-To-Image Generationcited by 7
  5. 2024CMC-Bench: Towards a New Paradigm of Visual Signal Compressioncited by 6
  6. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  7. 2025Qianfan-VL: Domain-Enhanced Universal Vision-Language Modelscited by 6
  8. 2026EvoGuard: An Extensible Agentic RL-based Framework for Practical and Evolving AI-Generated Image Detectioncited by 6
  9. 2025VARCO-VISION-2.0 Technical Reportcited by 5
  10. 2025Using GUI Agent for Electronic Design Automationcited by 5
  11. 2025Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoningcited by 4
  12. 2025Locate-Then-Examine: Grounded Region Reasoning Improves Detection of AI-Generated Imagescited by 4
  13. 2025Q-Save: Towards Scoring and Attribution for Generated Video Evaluationcited by 3
  14. 2025EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humanscited by 3
  15. 2024Towards Unified Benchmark and Models for Multi-Modal Perceptual Metricscited by 2
  16. 2024Embodied Image Quality Assessment for Robotic Intelligencecited by 2
  17. 2025OBIFormer: A Fast Attentive Denoising Framework for Oracle Bone Inscriptionscited by 2
  18. 2025Improve MLLM Benchmark Efficiency through Interviewcited by 2
  19. 2025The Ever-Evolving Science Examcited by 2
  20. 2024GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Modelscited by 1
  21. 2025Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Modelscited by 1
  22. 2025Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Headscited by 1
  23. 2025Who is a Better Player: LLM against LLMcited by 1
  24. 2026Towards Characterizing Scientific Image Utility and Upgradabilitycited by 1
  25. 2026GenScale: A Benchmark for Relative Object Scale in Image Generation and Editingcited by 1
  26. 2026Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessmentcited by 0
  27. 2026VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Visioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025M3-AGIQA: Multimodal, Multi-Round, Multi-Aspect AI-Generated Image Quality Assessment
  2. 2024AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM
  3. 2025Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content
  4. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
  5. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
  6. 2024DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks
  7. 2025AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images
  8. 2025What Makes a Good Generated Image? Investigating Human and Multimodal LLM Image Preference Alignment