# Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies

arXiv: https://arxiv.org/abs/2508.10523 · HTML: https://mlsanity.com/paper/2508.10523/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 218 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs](https://mlsanity.com/paper/2501.06186/) (2025)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Logical Neural Networks](https://mlsanity.com/paper/2006.13155/) (2020)
- [Large Language Models Can Learn Temporal Reasoning](https://mlsanity.com/paper/2401.06853/) (2024)
- [Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models](https://mlsanity.com/paper/2503.20752/) (2025)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models](https://mlsanity.com/paper/2410.23266/) (2024)
- [V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning](https://mlsanity.com/paper/2503.11495/) (2025)
- [From Understanding to Utilization: A Survey on Explainability for Large Language Models](https://mlsanity.com/paper/2401.12874/) (2024)
- [SocialIQA: Commonsense Reasoning about Social Interactions](https://mlsanity.com/paper/1904.09728/) (2019)
- [VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity](https://mlsanity.com/paper/2503.11557/) (2025)
- [Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios](https://mlsanity.com/paper/2501.04671/) (2025)
- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [A Survey on Compositional Generalization in Applications](https://mlsanity.com/paper/2302.01067/) (2023)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [EasyARC: Evaluating Vision Language Models on True Visual Reasoning](https://mlsanity.com/paper/2506.11595/) (2025)

## Built on by (most cited first, 17 of 17)

- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead](https://mlsanity.com/paper/2512.00903/) (2025)
- [Learning Vision-Language-Action World Models for Autonomous Driving](https://mlsanity.com/paper/2604.09059/) (2026)
- [Adaptive Detector-Verifier Framework for Zero-Shot Polyp Detection in Open-World Settings](https://mlsanity.com/paper/2512.12492/) (2025)
- [Mitigating Hallucination on Hallucination in RAG via Ensemble Voting](https://mlsanity.com/paper/2603.27253/) (2026)
- [An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation](https://mlsanity.com/paper/2606.00987/) (2026)
- [Why Relational Graphs Will Save the Next Generation of Vision Foundation Models?](https://mlsanity.com/paper/2508.18421/) (2025)
- [Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers](https://mlsanity.com/paper/2509.01161/) (2025)
- [A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture](https://mlsanity.com/paper/2509.01164/) (2025)
- [Adaptive Causal Coordination Detection for Social Media: A Memory-Guided Framework with Semi-Supervised Learning](https://mlsanity.com/paper/2601.00400/) (2026)
- [Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessment](https://mlsanity.com/paper/2601.09120/) (2026)
- [Conformal Risk Prediction for Non-Alcoholic Fatty Liver Disease Using Gradient Boosting with Distribution-Free Coverages](https://mlsanity.com/paper/2606.09860/) (2026)
- [VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference](https://mlsanity.com/paper/2606.14724/) (2026)
- [Bridging Geographic Bias in Urban Streetscape Inference via Lifelong Learning with Visual-Semantic Pivoting](https://mlsanity.com/paper/2606.15055/) (2026)
- [URDF Synthesis from RGB-D Sequences via Differentiable Joint Inference and Energy-Consistent Verification](https://mlsanity.com/paper/2606.18861/) (2026)
- [Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation](https://mlsanity.com/paper/2606.30647/) (2026)
- [Safe and Adaptive Cloud Healing: Verifying LLM-Generated Recovery Plans with a Neural-Symbolic World Model](https://mlsanity.com/paper/2607.01595/) (2026)

## Abstract (arXiv)

Visual reasoning matters for many computer vision tasks that go beyond surface-level object detection and classification. Despite progress in relational, symbolic, temporal, causal, and commonsense reasoning, existing surveys typically cover only one part of the problem, such as visual question answering, scene-graph generation, neuro-symbolic AI, or multimodal chain-of-thought, and rarely analyze reasoning types, methodologies, and evaluation protocols together. This survey addresses that gap. Following a structured literature review, we group visual reasoning into five major types (relational, symbolic, temporal, causal, and commonsense) and examine how each is implemented across methods that range from graph-based models, memory networks, attention mechanisms, and neuro-symbolic systems to reasoning with vision-language models (VLMs) and multimodal large language models (MLLMs), including visual chain-of-thought, visual programming, and tool-augmented and test-time reasoning. We then review evaluation protocols for functional correctness, structural consistency, and causal validity, and we analyze their limits in generalizability, reproducibility, faithfulness, and explanatory power. We also identify open challenges: scaling to complex scenes, integrating symbolic and neural paradigms more deeply, the shortage of comprehensive benchmarks, language-prior shortcuts and hallucination in foundation models, and reasoning under weak supervision. Finally, we set out a research agenda for vision systems and argue that connecting perception and reasoning is necessary for transparent, trustworthy, and cross-domain models, especially in high-stakes settings such as autonomous driving and medical diagnostics.