mlsanityData to 14 Oct 2026

"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models

arXiv 2502.11718 ↗First posted 17 Feb 2025cs.CL cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models' knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field. Our evaluation-friendly code and data have already been open-sourced.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 22

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
10 more
  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  2. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  3. 2024Measuring short-form factuality in large language modelscited by 373
  4. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  5. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  6. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  8. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 58
  9. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 43
  10. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 38
  2. 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Trainingcited by 17
  3. 2025CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Gamescited by 14

Similar, not cited

Close in wording, no citation either way

  1. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan
  2. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language
  3. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation
  4. 2025VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering
  5. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  6. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  7. 2026Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
  8. 2025Where do Large Vision-Language Models Look at when Answering Questions?