mlsanityData to 14 Oct 2026

AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?

arXiv 2410.21259 ↗First posted 28 Oct 2024cs.CV cs.AIcited by 21

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. However, the evaluation of LVLMs presents significant challenges as the evaluation benchmark always demands lots of human cost for its construction, and remains static, lacking flexibility once constructed. Even though automatic evaluation has been explored in textual modality, the visual modality remains under-explored. As a result, in this work, we address a question: "Can LVLMs themselves be used to benchmark each other in the visual automatically domain?". We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability. AutoBench-V leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. Through an extensive evaluation of nine popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 69 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

54 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  2. 2024RegionGPT: Towards Region Understanding Vision Language Model4 Mar
  3. 2022Deep Learning Approaches on Image Captioning: A Review31 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  13. 2019Object Detection in 20 Years: A Surveycited by 531
  14. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  15. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  16. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  17. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  18. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  19. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  20. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  21. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  22. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  23. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  24. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  25. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  26. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  27. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  28. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  29. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  30. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  31. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  32. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 47
  33. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 43
  34. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 41
  35. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 37
  36. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  37. 2024Task Me Anythingcited by 30
  38. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 7.

21 papers.

Sort:

By area:

  1. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 36
  2. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  3. 2024Piecing It All Together: Verifying Multi-Hop Multimodal Claimscited by 12
  4. 2024MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysiscited by 11
  5. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 11
  6. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  7. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scalecited by 8
  8. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  9. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  10. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  11. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  12. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
9 more
  1. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  2. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructionscited by 3
  3. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Eracited by 2
  4. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
  5. 2026Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMscited by 1
  6. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0
  7. 2026PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Modelscited by 0
  8. 2026REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluationcited by 0
  9. 2026ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specificationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  5. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language
  6. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  7. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
  8. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models