# Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

arXiv: https://arxiv.org/abs/2411.13754 · HTML: https://mlsanity.com/paper/2411.13754/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 88 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Learning to Compose Neural Networks for Question Answering](https://mlsanity.com/paper/1601.01705/) (2016)
- [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019)
- [An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2210.16773/) (2022)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)
- [Slot Abstractors: Toward Scalable Abstract Visual Reasoning](https://mlsanity.com/paper/2403.03458/) (2024)
- [AGQA 2.0: An Updated Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2204.06105/) (2022)

## Built on by (most cited first, 3 of 3)

- [IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&A](https://mlsanity.com/paper/2508.01984/) (2025)
- [DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes](https://mlsanity.com/paper/2505.03581/) (2025)
- [Neuro Symbolic Knowledge Reasoning for Procedural Video Question Answering](https://mlsanity.com/paper/2503.14957/) (2025)

## Abstract (arXiv)

Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios. Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query "determine the color of pen to the left of the child in red t-shirt sitting at the white table"). Meanwhile, its "parallel" computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.