# Understanding the computational demands underlying visual reasoning

arXiv: https://arxiv.org/abs/2108.03603 · HTML: https://mlsanity.com/paper/2108.03603/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 8 Aug 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning](https://mlsanity.com/paper/1611.05594/) (2016)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering](https://mlsanity.com/paper/1511.05234/) (2015)
- [Attention over learned object embeddings enables complex visual reasoning](https://mlsanity.com/paper/2012.08508/) (2020)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering](https://mlsanity.com/paper/1511.05960/) (2015)
- [Recurrent neural circuits for contour detection](https://mlsanity.com/paper/2010.15314/) (2020)
- [Deep Networks with Internal Selective Attention through Feedback Connections](https://mlsanity.com/paper/1407.3068/) (2014)
- [25 years of CNNs: Can we compare to human abstraction capabilities?](https://mlsanity.com/paper/1607.08366/) (2016)
- [Solving the Same-Different Task with Convolutional Neural Networks](https://mlsanity.com/paper/2101.09129/) (2021)
- [Evaluating the Progress of Deep Learning for Visual Relational Concepts](https://mlsanity.com/paper/2001.10857/) (2020)
- [Learning what and where to attend](https://mlsanity.com/paper/1805.08819/) (2018)
- [End-to-End Instance Segmentation with Recurrent Attention](https://mlsanity.com/paper/1605.09410/) (2016)
- [Program synthesis performance constrained by non-linear spatial relations in Synthetic Visual Reasoning Test](https://mlsanity.com/paper/1911.07721/) (2019)

## Built on by (most cited first, 13 of 13)

- [A Benchmark for Compositional Visual Reasoning](https://mlsanity.com/paper/2206.05379/) (2022)
- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [Systematic Visual Reasoning through Object-Centric Relational Abstraction](https://mlsanity.com/paper/2306.02500/) (2023)
- [Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention](https://mlsanity.com/paper/2303.01542/) (2023)
- [GAMR: A Guided Attention Model for (visual) Reasoning](https://mlsanity.com/paper/2206.04928/) (2022)
- [Slot Abstractors: Toward Scalable Abstract Visual Reasoning](https://mlsanity.com/paper/2403.03458/) (2024)
- [Recurrent Vision Transformer for Solving Visual Reasoning Problems](https://mlsanity.com/paper/2111.14576/) (2021)
- [Learning richness modulates equality reasoning in neural networks](https://mlsanity.com/paper/2503.09781/) (2025)
- [Conviformers: Convolutionally guided Vision Transformer](https://mlsanity.com/paper/2208.08900/) (2022)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approach](https://mlsanity.com/paper/2402.12675/) (2024)
- [Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning](https://mlsanity.com/paper/2309.13181/) (2023)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)

## Abstract (arXiv)

Visual understanding requires comprehending complex visual relations between objects within a scene. Here, we seek to characterize the computational demands for abstract visual reasoning. We do this by systematically assessing the ability of modern deep convolutional neural networks (CNNs) to learn to solve the "Synthetic Visual Reasoning Test" (SVRT) challenge, a collection of twenty-three visual reasoning problems. Our analysis reveals a novel taxonomy of visual reasoning tasks, which can be primarily explained by both the type of relations (same-different vs. spatial-relation judgments) and the number of relations used to compose the underlying rules. Prior cognitive neuroscience work suggests that attention plays a key role in humans' visual reasoning ability. To test this hypothesis, we extended the CNNs with spatial and feature-based attention mechanisms. In a second series of experiments, we evaluated the ability of these attention networks to learn to solve the SVRT challenge and found the resulting architectures to be much more efficient at solving the hardest of these visual reasoning tasks. Most importantly, the corresponding improvements on individual tasks partially explained our novel taxonomy. Overall, this work provides an granular computational account of visual reasoning and yields testable neuroscience predictions regarding the differential need for feature-based vs. spatial attention depending on the type of visual reasoning problem.