# ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

arXiv: https://arxiv.org/abs/2310.19301 · HTML: https://mlsanity.com/paper/2310.19301/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment](https://mlsanity.com/paper/2203.07190/) (2022)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?](https://mlsanity.com/paper/2307.02469/) (2023)
- [X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks](https://mlsanity.com/paper/2211.12402/) (2022)

## Built on by (most cited first, 27 of 27)

- [PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset](https://mlsanity.com/paper/2403.11116/) (2024)
- [VLind-Bench: Measuring Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2406.08702/) (2024)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation](https://mlsanity.com/paper/2410.20774/) (2024)
- [Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models](https://mlsanity.com/paper/2402.10884/) (2024)
- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering](https://mlsanity.com/paper/2503.18491/) (2025)
- [Evaluating Image Hallucination in Text-to-Image Generation with Question-Answering](https://mlsanity.com/paper/2409.12784/) (2024)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [WinClick: GUI Grounding with Multimodal Large Language Models](https://mlsanity.com/paper/2503.04730/) (2025)
- [Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble](https://mlsanity.com/paper/2501.13964/) (2025)
- [Synthesizing Visual Concepts as Vision-Language Programs](https://mlsanity.com/paper/2511.18964/) (2025)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation](https://mlsanity.com/paper/2601.07506/) (2026)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments](https://mlsanity.com/paper/2510.26006/) (2025)
- [NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?](https://mlsanity.com/paper/2411.17794/) (2024)
- [Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency](https://mlsanity.com/paper/2505.14405/) (2025)
- [MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts](https://mlsanity.com/paper/2403.14982/) (2024)
- [VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2404.16365/) (2024)
- [When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't](https://mlsanity.com/paper/2604.06422/) (2026)
- [Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models](https://mlsanity.com/paper/2604.12119/) (2026)
- [AI Judges in Design: Statistical Perspectives on Achieving Human Expert Equivalence With Vision-Language Models](https://mlsanity.com/paper/2504.00938/) (2025)
- [Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2607.26326/) (2026)
- [When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibration](https://mlsanity.com/paper/2607.29240/) (2026)
- [Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation](https://mlsanity.com/paper/2608.03691/) (2026)

## Abstract (arXiv)

Humans possess a strong capability for reasoning beyond common sense. For example, given an unconventional image of a goldfish laying on the table next to an empty fishbowl, a human would effortlessly determine that the fish is not inside the fishbowl. The case, however, may be different for a vision-language model, whose reasoning could gravitate towards the common scenario that the fish is inside the bowl, despite the visual input. In this paper, we introduce a novel probing dataset named ROME (reasoning beyond commonsense knowledge) to evaluate whether the state-of-the-art pre-trained vision-language models have the reasoning capability to correctly interpret counter-intuitive content. ROME contains images that defy commonsense knowledge with regards to color, shape, material, size and positional relation. Experiments on the state-of-the-art pre-trained vision-language models reveal that most of these models are still largely incapable of interpreting counter-intuitive scenarios. We hope that ROME will spur further investigations on reasoning beyond commonsense knowledge in vision-language research.