mlsanityData to 7 Oct 2026

A Surprising Failure? Multimodal LLMs and the NLVR Challenge

arXiv 2402.17793 ↗First posted 26 Feb 2024cs.AI cs.CL cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This study evaluates three state-of-the-art MLLMs -- GPT-4V, Gemini Pro, and the open-source model IDEFICS -- on the compositional natural language vision reasoning task NLVR. Given a human-written sentence paired with a synthetic image, this task requires the model to determine the truth value of the sentence with respect to the image. Despite the strong performance demonstrated by these models, we observe they perform poorly on NLVR, which was constructed to require compositional and spatial reasoning, and to be robust for semantic and systematic biases.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 10 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

7 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 6

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  4. 2017C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Datasetcited by 24
  5. 2018Visual Referring Expression Recognition: What Do Systems Actually Learn?cited by 20
  6. 2020What is Learned in Visually Grounded Neural Syntax Acquisitioncited by 15

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  2. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  3. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planningcited by 3
  4. 2024GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
  2. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  3. 2023An Examination of the Compositionality of Large Generative Vision-Language Models
  4. 2024Natural Language Inference Improves Compositionality in Vision-Language Models
  5. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  6. 2025ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
  7. 2026CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs
  8. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Data