mlsanityData to 7 Oct 2026

A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models

arXiv 2402.18409 ↗First posted 28 Feb 2024cs.AI cs.CL cs.CVcited by 6

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the Cookie Theft task in human cognitive tests, we propose a novel evaluation benchmark to evaluate high-level cognitive abilities of LVLMs using images with rich semantics. The benchmark consists of 251 images along with comprehensive annotations. It defines eight reasoning capabilities and comprises an image description task and a visual question answering task. Our evaluation of well-known LVLMs shows that there is still a significant gap in cognitive abilities between LVLMs and humans.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 55 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

21 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models8 Sep
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
7 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  5. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  6. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  7. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  3. 2024What Makes a Maze Look Like a Maze?cited by 11
  4. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  5. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  6. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
  2. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  3. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  6. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  7. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  8. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images