# Iterative Visual Reasoning Beyond Convolutions

arXiv: https://arxiv.org/abs/1803.11189 · HTML: https://mlsanity.com/paper/1803.11189/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2018.

## Built on by (most cited first, 30 of 38)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors](https://mlsanity.com/paper/2207.10761/) (2022)
- [Graph Convolutional Module for Temporal Action Localization in Videos](https://mlsanity.com/paper/2112.00302/) (2021)
- [Relation DETR: Exploring Explicit Position Relation Prior for Object Detection](https://mlsanity.com/paper/2407.11699/) (2024)
- [Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition](https://mlsanity.com/paper/2107.12090/) (2021)
- [Act Like a Radiologist: Towards Reliable Multi-view Correspondence Reasoning for Mammogram Mass Detection](https://mlsanity.com/paper/2105.10160/) (2021)
- [VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use](https://mlsanity.com/paper/2410.16400/) (2024)
- [Cross-receptive Focused Inference Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2207.02796/) (2022)
- [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022)
- [Improved GUI Grounding via Iterative Narrowing](https://mlsanity.com/paper/2411.13591/) (2024)
- [Towards the Unseen: Iterative Text Recognition by Distilling from Errors](https://mlsanity.com/paper/2107.12081/) (2021)
- [Beyond RGB: Scene-Property Synthesis with Neural Radiance Fields](https://mlsanity.com/paper/2206.04669/) (2022)
- [Instance-Level Relative Saliency Ranking with Graph Reasoning](https://mlsanity.com/paper/2107.03824/) (2021)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [L2C: Describing Visual Differences Needs Semantic Understanding of Individuals](https://mlsanity.com/paper/2102.01860/) (2021)
- [Learning Visual Models using a Knowledge Graph as a Trainer](https://mlsanity.com/paper/2102.08747/) (2021)
- [LP-DETR: Layer-wise Progressive Relations for Object Detection](https://mlsanity.com/paper/2502.05147/) (2025)
- [A Survey on Visual Transfer Learning using Knowledge Graphs](https://mlsanity.com/paper/2201.11794/) (2022)
- [Graphonomy: Universal Image Parsing via Graph Reasoning and Transfer](https://mlsanity.com/paper/2101.10620/) (2021)
- [Magic Layouts: Structural Prior for Component Detection in User Interface Designs](https://mlsanity.com/paper/2106.07615/) (2021)
- [Modeling Mask Uncertainty in Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2112.15362/) (2021)
- [Hybrid intelligence for dynamic job-shop scheduling with deep reinforcement learning and attention mechanism](https://mlsanity.com/paper/2201.00548/) (2022)
- [Language-Guided Diffusion Model for Visual Grounding](https://mlsanity.com/paper/2308.09599/) (2023)
- [Graph Neural Network Approach to Semantic Type Detection in Tables](https://mlsanity.com/paper/2405.00123/) (2024)
- [Learning to See the Elephant in the Room: Self-Supervised Context Reasoning in Humans and AI](https://mlsanity.com/paper/2211.12817/) (2022)
- [PRIMA: Planner-Reasoner Inside a Multi-task Reasoning Agent](https://mlsanity.com/paper/2202.00531/) (2022)
- [Analysis of Visual Reasoning on One-Stage Object Detection](https://mlsanity.com/paper/2202.13115/) (2022)
- [Transferring Dual Stochastic Graph Convolutional Network for Facial Micro-expression Recognition](https://mlsanity.com/paper/2203.05208/) (2022)
- [Exploring Contextual Relationships for Cervical Abnormal Cell Detection](https://mlsanity.com/paper/2207.04693/) (2022)

## Abstract (arXiv)

We present a novel framework for iterative visual reasoning. Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions. The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module. Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes. Both the local module and the global module roll-out iteratively and cross-feed predictions to each other to refine estimates. The final predictions are made by combining the best of both modules with an attention mechanism. We show strong performance over plain ConvNets, eg achieving an 8.4% absolute improvement on ADE measured by per-class average precision. Analysis also shows that the framework is resilient to missing regions for reasoning.