# Are Vision Transformers More Data Hungry Than Newborn Visual Systems?

arXiv: https://arxiv.org/abs/2312.02843 · HTML: https://mlsanity.com/paper/2312.02843/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 72 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Better plain ViT baselines for ImageNet-1k](https://mlsanity.com/paper/2205.01580/) (2022)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [The neuroconnectionist research programme](https://mlsanity.com/paper/2209.03718/) (2022)
- [Towards Data-Efficient Detection Transformers](https://mlsanity.com/paper/2203.09507/) (2022)
- [Training Vision Transformers with Only 2040 Images](https://mlsanity.com/paper/2201.10728/) (2022)
- [Time to augment self-supervised visual representation learning](https://mlsanity.com/paper/2207.13492/) (2022)
- [Deep Predictive Learning in Neocortex and Pulvinar](https://mlsanity.com/paper/2006.14800/) (2020)
- [Controlled-rearing studies of newborn chicks and deep neural networks](https://mlsanity.com/paper/2112.06106/) (2021)
- [Modeling Object Recognition in Newborn Chicks using Deep Neural Networks](https://mlsanity.com/paper/2106.07185/) (2021)

## Built on by (most cited first, 7 of 7)

- [Machine learning for modelling unstructured grid data in computational physics: a review](https://mlsanity.com/paper/2502.09346/) (2025)
- [Neural timescales from a computational perspective](https://mlsanity.com/paper/2409.02684/) (2024)
- [Human Gaze Boosts Object-Centered Representation Learning](https://mlsanity.com/paper/2501.02966/) (2025)
- [HexFormer: Hyperbolic Vision Transformer with Exponential Map Aggregation](https://mlsanity.com/paper/2601.19849/) (2026)
- [Toddlers' Active Gaze Behavior Supports Self-Supervised Object Learning](https://mlsanity.com/paper/2411.01969/) (2024)
- [Transformers self-organize like newborn visual systems when trained in prenatal worlds](https://mlsanity.com/paper/2601.03117/) (2026)
- [Temporal Slowness in Central Vision Drives Semantic Object Learning](https://mlsanity.com/paper/2602.04462/) (2026)

## Abstract (arXiv)

Vision transformers (ViTs) are top performing models on many computer vision benchmarks and can accurately predict human behavior on object recognition tasks. However, researchers question the value of using ViTs as models of biological learning because ViTs are thought to be more data hungry than brains, with ViTs requiring more training data to reach similar levels of performance. To test this assumption, we directly compared the learning abilities of ViTs and animals, by performing parallel controlled rearing experiments on ViTs and newborn chicks. We first raised chicks in impoverished visual environments containing a single object, then simulated the training data available in those environments by building virtual animal chambers in a video game engine. We recorded the first-person images acquired by agents moving through the virtual chambers and used those images to train self supervised ViTs that leverage time as a teaching signal, akin to biological visual systems. When ViTs were trained through the eyes of newborn chicks, the ViTs solved the same view invariant object recognition tasks as the chicks. Thus, ViTs were not more data hungry than newborn visual systems: both learned view invariant object representations in impoverished visual environments. The flexible and generic attention based learning mechanism in ViTs combined with the embodied data streams available to newborn animals appears sufficient to drive the development of animal-like object recognition.