# Teaching CORnet Human fMRI Representations for Enhanced Model-Brain Alignment

arXiv: https://arxiv.org/abs/2407.10414 · HTML: https://mlsanity.com/paper/2407.10414/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Learning From Brains How to Regularize Machines](https://mlsanity.com/paper/1911.05072/) (2019)
- [Towards robust vision by multi-task learning on monkey visual cortex](https://mlsanity.com/paper/2107.14344/) (2021)
- [A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains](https://mlsanity.com/paper/2310.13849/) (2023)
- [End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions](https://mlsanity.com/paper/2308.09431/) (2023)
- [Improving the Accuracy and Robustness of CNNs Using a Deep CCA Neural Data Regularizer](https://mlsanity.com/paper/2209.02582/) (2022)
- [Generate your neural signals from mine: individual-to-individual EEG converters](https://mlsanity.com/paper/2304.10736/) (2023)
- [Probing Human Visual Robustness with Neurally-Guided Deep Neural Networks](https://mlsanity.com/paper/2405.02564/) (2024)

## Built on by (most cited first, 2 of 2)

- [Language learning shapes visual category-selectivity in deep neural networks](https://mlsanity.com/paper/2502.16456/) (2025)
- [A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Data](https://mlsanity.com/paper/2601.12053/) (2026)

## Abstract (arXiv)

Deep convolutional neural networks (DCNNs) have demonstrated excellent performance in object recognition and have been found to share some similarities with brain visual processing. However, the substantial gap between DCNNs and human visual perception still exists. Functional magnetic resonance imaging (fMRI) as a widely used technique in cognitive neuroscience can record neural activation in the human visual cortex during the process of visual perception. Can we teach DCNNs human fMRI signals to achieve a more brain-like model? To answer this question, this study proposed ReAlnet-fMRI, a model based on the SOTA vision model CORnet but optimized using human fMRI data through a multi-layer encoding-based alignment framework. This framework has been shown to effectively enable the model to learn human brain representations. The fMRI-optimized ReAlnet-fMRI exhibited higher similarity to the human brain than both CORnet and the control model in within-and across-subject as well as within- and across-modality model-brain (fMRI and EEG) alignment evaluations. Additionally, we conducted an in-depth analyses to investigate how the internal representations of ReAlnet-fMRI differ from CORnet in encoding various object dimensions. These findings provide the possibility of enhancing the brain-likeness of visual models by integrating human neural data, helping to bridge the gap between computer vision and visual neuroscience.