# IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests arXiv: https://arxiv.org/abs/2505.12000 ยท HTML: https://mlsanity.com/paper/2505.12000/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 26 of its 33 references that are in the map. - First posted on arXiv 17 May 2025. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) - Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/) ## Builds on (most cited first, 26 of 26) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022) - [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020) - [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022) - [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023) - [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018) - [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022) - [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024) - [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022) - [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018) - [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017) - [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024) - [MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models](https://mlsanity.com/paper/2502.00698/) (2025) - [VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity](https://mlsanity.com/paper/2503.11557/) (2025) - [Image2Struct: Benchmarking Structure Extraction for Vision-Language Models](https://mlsanity.com/paper/2410.22456/) (2024) - [Generalized Planning for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2401.07426/) (2024) - [SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging](https://mlsanity.com/paper/2504.10642/) (2025) ## Built on by (most cited first, 5 of 5) - [Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models](https://mlsanity.com/paper/2508.12587/) (2025) - [Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles](https://mlsanity.com/paper/2601.13705/) (2026) - [The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?](https://mlsanity.com/paper/2508.21143/) (2025) - [Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs](https://mlsanity.com/paper/2604.16054/) (2026) - [StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning](https://mlsanity.com/paper/2606.00148/) (2026) ## Abstract (arXiv) Although large Vision-Language Models (VLMs) have demonstrated remarkable performance in a wide range of multimodal tasks, their true reasoning capabilities on human IQ tests remain underexplored. To advance research on the fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the reasoning capabilities of VLMs, which we argue are more important than the accuracy of the final prediction. **Our benchmark is visually centric, minimizing the dependence on unnecessary textual content**, thus encouraging models to derive answers primarily from image-based information rather than learned textual knowledge. To this end, we manually collected and annotated 500 visual IQ questions to **prevent unintentional data leakage during training**. Unlike prior work that focuses primarily on the accuracy of the final answer, we evaluate the reasoning ability of the models by assessing their explanations and the patterns used to solve each problem, along with the accuracy of the final prediction and human evaluation. Our experiments show that there are substantial performance disparities between tasks, with models such as `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest average accuracies of 0.615, 0.578, and 0.548, respectively. However, all models struggle with 3D spatial and anagram reasoning tasks, highlighting significant limitations in current VLMs' general reasoning abilities. In terms of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieved top averages of 0.696, 0.586, and 0.516, respectively. These results highlight inconsistencies between the reasoning processes of the models and their final answers, emphasizing the importance of evaluating the accuracy of the reasoning in addition to the final predictions.