mlsanityData to 14 Oct 2026

DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks

arXiv 2406.04470 ↗First posted 6 Jun 2024cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This study assesses the ability of Large Vision-Language Models (LVLMs) to differentiate between AI-generated and human-generated images. It introduces a new automated benchmark construction method for this evaluation. The experiment compared common LVLMs with human participants using a mixed dataset of AI and human-created images. Results showed that LVLMs could distinguish between the image types to some extent but exhibited a rightward bias, and perform significantly worse compared to humans. To build on these findings, we developed an automated benchmark construction process using AI. This process involved topic retrieval, narrative script generation, error embedding, and image generation, creating a diverse set of text-image pairs with intentional errors. We validated our method through constructing two caparable benchmarks. This study highlights the strengths and weaknesses of LVLMs in real-world understanding and advances benchmark construction techniques, providing a scalable and automatic approach for AI model evaluation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 33 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

25 of 33 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
13 more
  1. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  2. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  5. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  7. 2023GenImage: A Million-Scale Benchmark for Detecting AI-Generated Imagecited by 321
  8. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  9. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  10. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  11. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  12. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  13. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Modelscited by 13
  2. 2026THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  2. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  3. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  4. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
  5. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  6. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models
  7. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
  8. 2025Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models