# A Spectral Theory of Neural Prediction and Alignment

arXiv: https://arxiv.org/abs/2309.12821 · HTML: https://mlsanity.com/paper/2309.12821/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 93 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future](https://mlsanity.com/paper/2001.07092/) (2020)
- [Implicit Regularization of Random Feature Models](https://mlsanity.com/paper/2002.08404/) (2020)
- [Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization](https://mlsanity.com/paper/1805.10939/) (2018)

## Built on by (most cited first, 14 of 14)

- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Brain-Model Evaluations Need the NeuroAI Turing Test](https://mlsanity.com/paper/2502.16238/) (2025)
- [Neural population geometry and optimal coding of tasks with shared latent structure](https://mlsanity.com/paper/2402.16770/) (2024)
- [Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning](https://mlsanity.com/paper/2601.00791/) (2026)
- [Model-brain comparison using inter-animal transforms](https://mlsanity.com/paper/2510.02523/) (2025)
- [Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices](https://mlsanity.com/paper/2410.17998/) (2024)
- [Spectral Analysis of Representational Similarity with Limited Neurons](https://mlsanity.com/paper/2502.19648/) (2025)
- [Abstraction Induces the Brain Alignment of Language and Speech Models](https://mlsanity.com/paper/2602.04081/) (2026)
- [Large language models reorganize representational geometry during in-context learning](https://mlsanity.com/paper/2605.28854/) (2026)
- [Model alignment using inter-modal bridges](https://mlsanity.com/paper/2505.12322/) (2025)
- [Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models](https://mlsanity.com/paper/2509.23333/) (2025)
- [On Kernel Eigen-alignments of KRR: Reconstruction and Generalization](https://mlsanity.com/paper/2605.15240/) (2026)
- [Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment](https://mlsanity.com/paper/2605.20127/) (2026)

## Abstract (arXiv)

The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.