# CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

arXiv: https://arxiv.org/abs/2603.27958 · HTML: https://mlsanity.com/paper/2603.27958/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models](https://mlsanity.com/paper/2407.17773/) (2024)
- [Multimodal Analogical Reasoning over Knowledge Graphs](https://mlsanity.com/paper/2210.00312/) (2022)
- [FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks](https://mlsanity.com/paper/2501.05396/) (2025)
- [The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks](https://mlsanity.com/paper/2410.07391/) (2024)
- [Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models](https://mlsanity.com/paper/2510.07632/) (2025)
- [VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning](https://mlsanity.com/paper/2503.00043/) (2025)
- [Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models](https://mlsanity.com/paper/2505.20021/) (2025)
- [Compositional Scene Understanding through Inverse Generative Modeling](https://mlsanity.com/paper/2505.21780/) (2025)
- [What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives](https://mlsanity.com/paper/2510.03075/) (2025)
- [Composition-Grounded Data Synthesis for Visual Reasoning](https://mlsanity.com/paper/2510.15040/) (2025)
- [Can Multimodal Large Language Model Think Analogically?](https://mlsanity.com/paper/2411.01307/) (2024)

## Built on by (most cited first, 1 of 1)

- [PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogies](https://mlsanity.com/paper/2610.01383/) (2026)

## Abstract (arXiv)

Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.