# Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex

arXiv: https://arxiv.org/abs/2406.16935 · HTML: https://mlsanity.com/paper/2406.16935/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 84 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning to Learn Single Domain Generalization](https://mlsanity.com/paper/2003.13216/) (2020)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Robust and Generalizable Visual Representation Learning via Random Convolutions](https://mlsanity.com/paper/2007.13003/) (2020)
- [Learning to Balance Specificity and Invariance for In and Out of Domain Generalization](https://mlsanity.com/paper/2008.12839/) (2020)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness](https://mlsanity.com/paper/2102.11273/) (2021)

## Built on by (most cited first, 3 of 3)

- [Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations](https://mlsanity.com/paper/2412.09115/) (2024)
- [Omni-iEEG: A Large-Scale, Comprehensive iEEG Dataset and Benchmark for Epilepsy Research](https://mlsanity.com/paper/2602.16072/) (2026)
- [STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex](https://mlsanity.com/paper/2607.15631/) (2026)

## Abstract (arXiv)

We characterized the generalization capabilities of DNN-based encoding models when predicting neuronal responses from the visual cortex. We collected textitMacaqueITBench, a large-scale dataset of neural population responses from the macaque inferior temporal (IT) cortex to over 300,000 images, comprising 8,233 unique natural images presented to seven monkeys over 109 sessions. Using textitMacaqueITBench, we investigated the impact of distribution shifts on models predicting neural activity by dividing the images into Out-Of-Distribution (OOD) train and test splits. The OOD splits included several different image-computable types including image contrast, hue, intensity, temperature, and saturation. Compared to the performance on in-distribution test images -- the conventional way these models have been evaluated -- models performed worse at predicting neuronal responses to out-of-distribution images, retaining as little as 20% of the performance on in-distribution test images. The generalization performance under OOD shifts can be well accounted by a simple image similarity metric -- the cosine distance between image representations extracted from a pre-trained object recognition model is a strong predictor of neural predictivity under different distribution shifts. The dataset of images, neuronal firing rate recordings, and computational benchmarks are hosted publicly at: https://bit.ly/3zeutVd.