# Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems

arXiv: https://arxiv.org/abs/2411.01173 · HTML: https://mlsanity.com/paper/2411.01173/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 91 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [Solving Bongard Problems with a Visual Language and Pragmatic Reasoning](https://mlsanity.com/paper/1804.04452/) (2018)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [Evaluating Understanding on Conceptual Abstraction Benchmarks](https://mlsanity.com/paper/2206.14187/) (2022)
- [Visual analogy: Deep learning versus compositional models](https://mlsanity.com/paper/2105.07065/) (2021)
- [MindSet: Vision. A toolbox for testing DNNs on key psychological experiments](https://mlsanity.com/paper/2404.05290/) (2024)
- [Using Program Synthesis and Inductive Logic Programming to solve Bongard Problems](https://mlsanity.com/paper/2110.09947/) (2021)
- [The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks](https://mlsanity.com/paper/2410.07391/) (2024)

## Built on by (most cited first, 15 of 15)

- [Do AI Models Perform Human-like Abstract Reasoning Across Modalities?](https://mlsanity.com/paper/2510.02125/) (2025)
- [SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks](https://mlsanity.com/paper/2506.14512/) (2025)
- [Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems](https://mlsanity.com/paper/2508.12026/) (2025)
- [Toward Cognitive Supersensing in Multimodal Large Language Model](https://mlsanity.com/paper/2602.01541/) (2026)
- [Synthesizing Visual Concepts as Vision-Language Programs](https://mlsanity.com/paper/2511.18964/) (2025)
- [Beyond Text: Implementing Multimodal Large Language Model-Powered Multi-Agent Systems Using a No-Code Platform](https://mlsanity.com/paper/2501.00750/) (2025)
- [Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models](https://mlsanity.com/paper/2511.17487/) (2025)
- [SPHINX: A Synthetic Environment for Visual Perception and Reasoning](https://mlsanity.com/paper/2511.20814/) (2025)
- [Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles](https://mlsanity.com/paper/2601.13705/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction](https://mlsanity.com/paper/2607.08233/) (2026)
- [World-Time Compute with Verified Code World Models](https://mlsanity.com/paper/2609.09163/) (2026)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [Neurophysiological Characteristics of Adaptive Reasoning for Creative Problem-Solving Strategy](https://mlsanity.com/paper/2511.07912/) (2025)
- [GenPlanner: From Noise to Plans -- Emergent Reasoning in Flow Matching and Diffusion Models](https://mlsanity.com/paper/2602.18812/) (2026)

## Abstract (arXiv)

Abstract visual reasoning (AVR) involves discovering shared concepts across images through analogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal description. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited solution strategies and testing 4 proprietary and 4 open-access models on 3 BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard-OpenWorld) images. Despite some successes on real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-RWR, a dataset representing synthetic BP concepts using real-world images. Our findings suggest that weak MLLM performance on classical BPs is not due to the domain specificity, but rather comes from their general AVR limitations. Code and dataset are available at: https://github.com/pavonism/bongard-rwr