# Tracking Without Re-recognition in Humans and Machines

arXiv: https://arxiv.org/abs/2105.13351 · HTML: https://mlsanity.com/paper/2105.13351/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 85 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [Know Your Surroundings: Exploiting Scene Information for Object Tracking](https://mlsanity.com/paper/2003.11014/) (2020)
- [Learning long-range spatial dependencies with horizontal gated-recurrent units](https://mlsanity.com/paper/1805.08315/) (2018)
- [Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency](https://mlsanity.com/paper/2006.16736/) (2020)
- [Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning](https://mlsanity.com/paper/2010.00763/) (2020)
- [Disentangling neural mechanisms for perceptual grouping](https://mlsanity.com/paper/1906.01558/) (2019)
- [Recurrent neural circuits for contour detection](https://mlsanity.com/paper/2010.15314/) (2020)
- [Stable and expressive recurrent vision models](https://mlsanity.com/paper/2005.11362/) (2020)
- [How Deep is the Feature Analysis underlying Rapid Visual Categorization?](https://mlsanity.com/paper/1606.01167/) (2016)
- [Hierarchical Attentive Recurrent Tracking](https://mlsanity.com/paper/1706.09262/) (2017)

## Built on by (most cited first, 12 of 12)

- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Learning What and Where: Disentangling Location and Identity Tracking Without Supervision](https://mlsanity.com/paper/2205.13349/) (2022)
- [The 3D-PC: a benchmark for visual perspective taking in humans and machines](https://mlsanity.com/paper/2406.04138/) (2024)
- [Understanding Visual Feature Reliance through the Lens of Complexity](https://mlsanity.com/paper/2407.06076/) (2024)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [Computing a human-like reaction time metric from stable recurrent vision models](https://mlsanity.com/paper/2306.11582/) (2023)
- [Extreme Image Transformations Affect Humans and Machines Differently](https://mlsanity.com/paper/2212.13967/) (2022)
- [Robustness of Humans and Machines on Object Recognition with Extreme Image Transformations](https://mlsanity.com/paper/2205.05167/) (2022)
- [Tracking objects that change in appearance with phase synchrony](https://mlsanity.com/paper/2410.02094/) (2024)
- [Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning](https://mlsanity.com/paper/2309.13181/) (2023)
- [Extreme Image Transformations Facilitate Robust Latent Object Representations](https://mlsanity.com/paper/2310.07725/) (2023)

## Abstract (arXiv)

Imagine trying to track one particular fruitfly in a swarm of hundreds. Higher biological visual systems have evolved to track moving objects by relying on both appearance and motion features. We investigate if state-of-the-art deep neural networks for visual tracking are capable of the same. For this, we introduce PathTracker, a synthetic visual challenge that asks human observers and machines to track a target object in the midst of identical-looking "distractor" objects. While humans effortlessly learn PathTracker and generalize to systematic variations in task design, state-of-the-art deep networks struggle. To address this limitation, we identify and model circuit mechanisms in biological brains that are implicated in tracking objects based on motion cues. When instantiated as a recurrent network, our circuit model learns to solve PathTracker with a robust visual strategy that rivals human performance and explains a significant proportion of their decision-making on the challenge. We also show that the success of this circuit model extends to object tracking in natural videos. Adding it to a transformer-based architecture for object tracking builds tolerance to visual nuisances that affect object appearance, resulting in a new state-of-the-art performance on the large-scale TrackingNet object tracking challenge. Our work highlights the importance of building artificial vision models that can help us better understand human vision and improve computer vision.