Data to 7 Oct 2026

ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

arXiv 2310.19301 ↗First posted 30 Oct 2023cs.CL cs.AI cs.CVcited by 27

On the map: LLM reasoning → Multimodal reasoning

also in 1 more

Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Humans possess a strong capability for reasoning beyond common sense. For example, given an unconventional image of a goldfish laying on the table next to an empty fishbowl, a human would effortlessly determine that the fish is not inside the fishbowl. The case, however, may be different for a vision-language model, whose reasoning could gravitate towards the common scenario that the fish is inside the bowl, despite the visual input. In this paper, we introduce a novel probing dataset named ROME (reasoning beyond commonsense knowledge) to evaluate whether the state-of-the-art pre-trained vision-language models have the reasoning capability to correctly interpret counter-intuitive content. ROME contains images that defy commonsense knowledge with regards to color, shape, material, size and positional relation. Experiments on the state-of-the-art pre-trained vision-language models reveal that most of these models are still largely incapable of interpreting counter-intuitive scenarios. We hope that ROME will spur further investigations on reasoning beyond commonsense knowledge in vision-language research.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 24 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

10 of 24 references.

References · 10

2023Visual Instruction Tuningcited by 7,792
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 7; 2026: 7.

27 papers.

Sort:

By area:

2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
2026Large Language Model Reasoning Failurescited by 20
2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
2024Evaluating Image Hallucination in Text-to-Image Generation with Question-Answeringcited by 15
2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
2025Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumblecited by 9

15 more

2025Synthesizing Visual Concepts as Vision-Language Programscited by 6
2024Benchmarking VLMs' Reasoning About Persuasive Atypical Imagescited by 5
2024Foundation Models for Autonomous Robots in Unstructured Environmentscited by 3
2025CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environmentscited by 3
2026Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluationcited by 3
2024NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?cited by 2
2025Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistencycited by 2
2024MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughtscited by 1
2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
2026When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don'tcited by 1
2025AI Judges in Design: Statistical Perspectives on Achieving Human Expert Equivalence With Vision-Language Modelscited by 0
2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Modelscited by 0
2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
2026When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibrationcited by 0
2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way