# The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes

arXiv: https://arxiv.org/abs/2301.03198 · HTML: https://mlsanity.com/paper/2301.03198/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 11 of 11)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Learning From Brains How to Regularize Machines](https://mlsanity.com/paper/1911.05072/) (2019)
- [Towards robust vision by multi-task learning on monkey visual cortex](https://mlsanity.com/paper/2107.14344/) (2021)
- [The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion](https://mlsanity.com/paper/2104.13714/) (2021)
- [The Sensorium competition on predicting large-scale mouse primary visual cortex activity](https://mlsanity.com/paper/2206.08666/) (2022)
- [The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence](https://mlsanity.com/paper/1905.05675/) (2019)

## Built on by (most cited first, 30 of 36)

- [fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding](https://mlsanity.com/paper/2311.00342/) (2023)
- [NeuralBench: A Unifying Framework to Benchmark NeuroAI Models](https://mlsanity.com/paper/2605.08495/) (2026)
- [The Wisdom of a Crowd of Brains: A Universal Brain Encoder](https://mlsanity.com/paper/2406.12179/) (2024)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [Brain Decodes Deep Nets](https://mlsanity.com/paper/2312.01280/) (2023)
- [The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solution](https://mlsanity.com/paper/2308.00262/) (2023)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion](https://mlsanity.com/paper/2411.09723/) (2024)
- [SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning](https://mlsanity.com/paper/2508.10298/) (2025)
- [Joint fMRI Decoding and Encoding with Latent Embedding Alignment](https://mlsanity.com/paper/2303.14730/) (2023)
- [Memory Encoding Model](https://mlsanity.com/paper/2308.01175/) (2023)
- [BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning](https://mlsanity.com/paper/2405.18808/) (2024)
- [AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space](https://mlsanity.com/paper/2406.18344/) (2024)
- [Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage](https://mlsanity.com/paper/2510.17833/) (2025)
- [Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer](https://mlsanity.com/paper/2510.25976/) (2025)
- [BrainExplore: Large-Scale Discovery of Interpretable Visual Representations in the Human Brain](https://mlsanity.com/paper/2512.08560/) (2025)
- [Applicability of scaling laws to vision encoding models](https://mlsanity.com/paper/2308.00678/) (2023)
- [Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI](https://mlsanity.com/paper/2312.00236/) (2023)
- [A Parameter-efficient Multi-subject Model for Predicting fMRI Activity](https://mlsanity.com/paper/2308.02351/) (2023)
- [Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex](https://mlsanity.com/paper/2309.15018/) (2023)
- [Aligned with LLM: a new multi-modal training paradigm for encoding fMRI activity in visual cortex](https://mlsanity.com/paper/2401.03851/) (2024)
- [NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework](https://mlsanity.com/paper/2408.14950/) (2024)
- [Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures](https://mlsanity.com/paper/2309.09983/) (2023)
- [Bayesian Comparisons Between Representations](https://mlsanity.com/paper/2411.08739/) (2024)
- [Brain-like emergent properties in deep networks: impact of network architecture, datasets and training](https://mlsanity.com/paper/2411.16326/) (2024)
- [The ISLab Solution to the Algonauts Challenge 2025: A Multimodal Deep Learning Approach to Brain Response Prediction](https://mlsanity.com/paper/2508.06499/) (2025)
- [MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding](https://mlsanity.com/paper/2605.29850/) (2026)
- [Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain](https://mlsanity.com/paper/2404.18772/) (2024)
- [Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli](https://mlsanity.com/paper/2410.04497/) (2024)
- [Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex](https://mlsanity.com/paper/2410.06067/) (2024)

## Abstract (arXiv)

The sciences of biological and artificial intelligence are ever more intertwined. Neural computational principles inspire new intelligent machines, which are in turn used to advance theoretical understanding of the brain. To promote further exchange of ideas and collaboration between biological and artificial intelligence researchers, we introduce the 2023 installment of the Algonauts Project challenge: How the Human Brain Makes Sense of Natural Scenes (http://algonauts.csail.mit.edu). This installment prompts the fields of artificial and biological intelligence to come together towards building computational models of the visual brain using the largest and richest dataset of fMRI responses to visual scenes, the Natural Scenes Dataset (NSD). NSD provides high-quality fMRI responses to ~73,000 different naturalistic colored scenes, making it the ideal candidate for data-driven model building approaches promoted by the 2023 challenge. The challenge is open to all and makes results directly comparable and transparent through a public leaderboard automatically updated after each submission, thus allowing for rapid model development. We believe that the 2023 installment will spark symbiotic collaborations between biological and artificial intelligence scientists, leading to a deeper understanding of the brain through cutting-edge computational models and to novel ways of engineering artificial intelligent agents through inductive biases from biological systems.