# Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans

arXiv: https://arxiv.org/abs/2306.00294 · HTML: https://mlsanity.com/paper/2306.00294/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Disentangling neural mechanisms for perceptual grouping](https://mlsanity.com/paper/1906.01558/) (2019)
- [Capturing the objects of vision with neural networks](https://mlsanity.com/paper/2109.03351/) (2021)
- [Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention](https://mlsanity.com/paper/2303.01542/) (2023)
- [Unsupervised Segmentation in Real-World Images via Spelke Object Inference](https://mlsanity.com/paper/2205.08515/) (2022)

## Built on by (most cited first, 5 of 5)

- [Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?](https://mlsanity.com/paper/2510.24709/) (2025)
- [Dissecting Query-Key Interaction in Vision Transformers](https://mlsanity.com/paper/2405.14880/) (2024)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approach](https://mlsanity.com/paper/2402.12675/) (2024)
- [Generating metamers of human scene understanding](https://mlsanity.com/paper/2601.11675/) (2026)

## Abstract (arXiv)

The spreading of attention has been proposed as a mechanism for how humans group features to segment objects. However, such a mechanism has not yet been implemented and tested in naturalistic images. Here, we leverage the feature maps from self-supervised vision Transformers and propose a model of human object-based attention spreading and segmentation. Attention spreads within an object through the feature affinity signal between different patches of the image. We also collected behavioral data on people grouping objects in natural images by judging whether two dots are on the same object or on two different objects. We found that our models of affinity spread that were built on feature maps from the self-supervised Transformers showed significant improvement over baseline and CNN based models on predicting reaction time patterns of humans, despite not being trained on the task or with any other object labels. Our work provides new benchmarks for evaluating models of visual representation learning including Transformers.