# A Surprising Failure? Multimodal LLMs and the NLVR Challenge

arXiv: https://arxiv.org/abs/2402.17793 · HTML: https://mlsanity.com/paper/2402.17793/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents](https://mlsanity.com/paper/2306.16527/) (2023)
- [C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset](https://mlsanity.com/paper/1704.08243/) (2017)
- [Visual Referring Expression Recognition: What Do Systems Actually Learn?](https://mlsanity.com/paper/1805.11818/) (2018)
- [What is Learned in Visually Grounded Neural Syntax Acquisition](https://mlsanity.com/paper/2005.01678/) (2020)

## Built on by (most cited first, 4 of 4)

- [KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models](https://mlsanity.com/paper/2407.17773/) (2024)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning](https://mlsanity.com/paper/2506.22992/) (2025)
- [GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Models](https://mlsanity.com/paper/2408.05894/) (2024)

## Abstract (arXiv)

This study evaluates three state-of-the-art MLLMs -- GPT-4V, Gemini Pro, and the open-source model IDEFICS -- on the compositional natural language vision reasoning task NLVR. Given a human-written sentence paired with a synthetic image, this task requires the model to determine the truth value of the sentence with respect to the image. Despite the strong performance demonstrated by these models, we observe they perform poorly on NLVR, which was constructed to require compositional and spatial reasoning, and to be robust for semantic and systematic biases.