# A Dataset and Architecture for Visual Reasoning with a Working Memory

arXiv: https://arxiv.org/abs/1803.06092 · HTML: https://mlsanity.com/paper/1803.06092/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2018.

## Built on by (most cited first, 12 of 12)

- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)
- [Weakly Supervised Temporal Adjacent Network for Language Grounding](https://mlsanity.com/paper/2106.16136/) (2021)
- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)
- [Hopper: Multi-hop Transformer for Spatiotemporal Reasoning](https://mlsanity.com/paper/2103.10574/) (2021)
- [Solving the Same-Different Task with Convolutional Neural Networks](https://mlsanity.com/paper/2101.09129/) (2021)
- [Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs](https://mlsanity.com/paper/2504.02789/) (2025)
- [Quantifying artificial intelligence through algorithmic generalization](https://mlsanity.com/paper/2411.05943/) (2024)
- [Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding](https://mlsanity.com/paper/2210.11933/) (2022)
- [IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation](https://mlsanity.com/paper/2307.06698/) (2023)
- [IWISDM: Assessing instruction following in multimodal models at scale](https://mlsanity.com/paper/2406.14343/) (2024)
- [Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation](https://mlsanity.com/paper/2303.05983/) (2023)

## Abstract (arXiv)

A vexing problem in artificial intelligence is reasoning about events that occur in complex, changing visual stimuli such as in video analysis or game play. Inspired by a rich tradition of visual reasoning and memory in cognitive psychology and neuroscience, we developed an artificial, configurable visual question and answer dataset (COG) to parallel experiments in humans and animals. COG is much simpler than the general problem of video analysis, yet it addresses many of the problems relating to visual and logical reasoning and memory -- problems that remain challenging for modern deep learning architectures. We additionally propose a deep learning architecture that performs competitively on other diagnostic VQA datasets (i.e. CLEVR) as well as easy settings of the COG dataset. However, several settings of COG result in datasets that are progressively more challenging to learn. After training, the network can zero-shot generalize to many new tasks. Preliminary analyses of the network architectures trained on COG demonstrate that the network accomplishes the task in a manner interpretable to humans.