mlsanityData to 7 Oct 2026

SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models

arXiv 2502.13059 ↗First posted 18 Feb 2025cs.CLcited by 61

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly their ability to produce content grounded in factual information (e.g. common and domain-specific knowledge). In this work, we introduce SimpleVQA, the first comprehensive multi-modal benchmark to evaluate the factuality ability of MLLMs to answer natural language short questions. SimpleVQA is characterized by six key features: it covers multiple tasks and multiple scenarios, ensures high quality and challenging queries, maintains static and timeless reference answers, and is straightforward to evaluate. Our approach involves categorizing visual question-answering items into 9 different tasks around objective events or common knowledge and situating these within 9 topics. Rigorous quality control processes are implemented to guarantee high-quality, concise, and clear answers, facilitating evaluation with minimal variance via an LLM-as-a-judge scoring system. Using SimpleVQA, we perform a comprehensive assessment of leading 18 MLLMs and 8 text-only LLMs, delving into their image comprehension and text generation abilities by identifying and analyzing error cases.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 42 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

34 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
19 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  5. 2024Measuring short-form factuality in large language modelscited by 345
  6. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  7. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  8. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  9. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  10. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  11. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  12. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 39
  13. 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33
  14. 2024SciMMIR: Benchmarking Scientific Multi-modal Information Retrievalcited by 22
  15. 2024Evaluating and Aligning CodeLLMs on Human Preferencecited by 22
  16. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 19
  17. 2024Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Modelscited by 6
  18. 2024SVIPTR: Fast and Efficient Scene Text Recognition with Vision Permutable Extractorcited by 5
  19. 2024XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parsercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 44.

61 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2026Qwen3.5-Omni Technical Reportcited by 146
  4. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  5. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  6. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  7. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  8. 2025Kwai Keye-VL Technical Reportcited by 47
  9. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  10. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  11. 2025Step-GUI Technical Reportcited by 38
  12. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 37
38 more
  1. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  2. 2026STEP3-VL-10B Technical Reportcited by 28
  3. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  4. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 18
  5. 2026OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agentscited by 18
  6. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 12
  7. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  8. 2026ERNIE 5.0 Technical Reportcited by 9
  9. 2026Context Unrolling in Omni Modelscited by 8
  10. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  11. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMscited by 6
  12. 2025MindWatcher: Toward Smarter Multimodal Tool-Integrated Reasoningcited by 6
  13. 2026Memory Intelligence Agentcited by 6
  14. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scalescited by 5
  15. 2026WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Modelscited by 5
  16. 2026BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agentscited by 5
  17. 2026ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewardscited by 4
  18. 2025AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluationcited by 3
  19. 2025CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Modelscited by 3
  20. 2026ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparisoncited by 3
  21. 2026DeepInsight: A Unified Evaluation Infrastructure Across the Physical AI Stackcited by 3
  22. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Modelscited by 3
  23. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgmentscited by 2
  24. 2026Reinforcement Learning with Robust Rubric Rewardscited by 2
  25. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  26. 2026Scaling Native Multimodal Pre-Training From Scratchcited by 2
  27. 2026TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streamingcited by 2
  28. 2026ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agentscited by 1
  29. 2026EduVQA: Towards Concept-Aware Assessment of Educational AI-Generated Videoscited by 1
  30. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraintscited by 1
  31. 2026DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agentscited by 1
  32. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
  33. 2026SEED: Targeted Data Selection by Weighted Independent Setcited by 1
  34. 2026ViQ: Text-Aligned Visual Quantized Representations at Any Resolutioncited by 1
  35. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 1
  36. 2026Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learningcited by 1
  37. 2025SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasetscited by 0
  38. 2026RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-trainingcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering
  2. 2025Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models
  3. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
  4. 2025Evaluating Variance in Visual Question Answering Benchmarks
  5. 2024Towards Flexible Evaluation for Generative Visual Question Answering
  6. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  7. 2024Revisiting Multi-Modal LLM Evaluation
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models