# Capturing the objects of vision with neural networks

arXiv: https://arxiv.org/abs/2109.03351 · HTML: https://mlsanity.com/paper/2109.03351/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 329 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 30 of 52)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [Interpreting Deep Visual Representations via Network Dissection](https://mlsanity.com/paper/1711.05611/) (2017)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)

## Built on by (most cited first, 21 of 21)

- [Breaking the "Object" in Video Object Segmentation](https://mlsanity.com/paper/2212.06200/) (2022)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)
- [Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?](https://mlsanity.com/paper/2510.24709/) (2025)
- [Self-supervised Amodal Video Object Segmentation](https://mlsanity.com/paper/2210.12733/) (2022)
- [Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?](https://mlsanity.com/paper/2305.17023/) (2023)
- [Object-based active inference](https://mlsanity.com/paper/2209.01258/) (2022)
- [Object-Centric Pretraining via Target Encoder Bootstrapping](https://mlsanity.com/paper/2503.15141/) (2025)
- [Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans](https://mlsanity.com/paper/2306.00294/) (2023)
- [Regularity as Intrinsic Reward for Free Play](https://mlsanity.com/paper/2312.01473/) (2023)
- [Object-centric proto-symbolic behavioural reasoning from pixels](https://mlsanity.com/paper/2411.17438/) (2024)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [BIFRNet: A Brain-Inspired Feature Restoration DNN for Partially Occluded Image Recognition](https://mlsanity.com/paper/2303.01309/) (2023)
- [Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks](https://mlsanity.com/paper/2306.01870/) (2023)
- [A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes](https://mlsanity.com/paper/2408.01322/) (2024)
- [Human-like Object Grouping in Self-supervised Vision Transformers](https://mlsanity.com/paper/2603.13994/) (2026)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)
- [Lightweight, error-tolerant edge detection using memristor-enabled stochastic logics](https://mlsanity.com/paper/2402.16908/) (2024)
- [Ideal Observer for Segmentation of Dead Leaves Images](https://mlsanity.com/paper/2512.05539/) (2025)

## Abstract (arXiv)

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked, and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioral studies have documented how object representations emerge through grouping, amodal completion, proto-objects, and object files. Deep neural network (DNN) models of visual object recognition, by contrast, remain largely tethered to the sensory input, despite achieving human-level performance at labeling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving development of deep neural network models that will put the object into object recognition.