# Learning to See the Elephant in the Room: Self-Supervised Context Reasoning in Humans and AI

arXiv: https://arxiv.org/abs/2211.12817 · HTML: https://mlsanity.com/paper/2211.12817/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 104 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Unsupervised Object-Level Representation Learning from Scene Images](https://mlsanity.com/paper/2106.11952/) (2021)
- [Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships](https://mlsanity.com/paper/1807.00119/) (2018)
- [Iterative Visual Reasoning Beyond Convolutions](https://mlsanity.com/paper/1803.11189/) (2018)
- [Putting visual object recognition in context](https://mlsanity.com/paper/1911.07349/) (2019)
- [Finding any Waldo: zero-shot invariant and efficient visual search](https://mlsanity.com/paper/1807.10587/) (2018)
- [Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition](https://mlsanity.com/paper/1511.04196/) (2015)
- [When Pigs Fly: Contextual Reasoning in Synthetic and Natural Scenes](https://mlsanity.com/paper/2104.02215/) (2021)
- [Learning Structured Inference Neural Networks with Label Relations](https://mlsanity.com/paper/1511.05616/) (2015)
- [Visual Search Asymmetry: Deep Nets and Humans Share Similar Inherent Biases](https://mlsanity.com/paper/2106.02953/) (2021)

## Built on by (most cited first, 2 of 2)

- [Adaptive Contextual Perception: How to Generalize to New Backgrounds and Ambiguous Objects](https://mlsanity.com/paper/2306.05963/) (2023)
- [Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning](https://mlsanity.com/paper/2607.10678/) (2026)

## Abstract (arXiv)

Humans rarely perceive objects in isolation but interpret scenes through relationships among co-occurring elements. How such contextual knowledge is acquired without explicit supervision remains unclear. Here we combine human psychophysics experiments with computational modelling to study the emergence of contextual reasoning. Participants were exposed to novel objects embedded in naturalistic scenes that followed predefined contextual rules capturing global context, local context and crowding. After viewing short training videos, participants completed a "lift-the-flap" task in which a hidden object had to be inferred from the surrounding context under variations in size, resolution and spatial arrangement. Humans rapidly learned these contextual associations without labels or feedback and generalised robustly across contextual changes. We then introduce SeCo (Self-supervised learning for Context Reasoning), a biologically inspired model that learns contextual relationships from complex scenes. SeCo encodes targets and context with separate vision encoders and stores latent contextual priors in a learnable external memory module. Given contextual cues, the model retrieves likely object representations to infer hidden targets. SeCo outperforms state-of-the-art self-supervised learning approaches and predicts object placements most consistent with human behaviour, highlighting the central role of contextual associations in scene understanding.