# Exploring the Properties and Evolution of Neural Network Eigenspaces during Training

arXiv: https://arxiv.org/abs/2106.09526 · HTML: https://mlsanity.com/paper/2106.09526/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 8 of 8)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Feature Space Saturation during Training](https://mlsanity.com/paper/2006.08679/) (2020)
- [Spectral Analysis of Latent Representations](https://mlsanity.com/paper/1907.08589/) (2019)

## Built on by (most cited first, 1 of 1)

- [Receptive Field Refinement for Convolutional Neural Networks Reliably Improves Predictive Performance](https://mlsanity.com/paper/2211.14487/) (2022)

## Abstract (arXiv)

In this work we explore the information processing inside neural networks using logistic regression probes citeprobes and the saturation metric citefeaturespacesaturation. We show that problem difficulty and neural network capacity affect the predictive performance in an antagonistic manner, opening the possibility of detecting over- and under-parameterization of neural networks for a given task. We further show that the observed effects are independent from previously reported pathological patterns like the ``tail pattern'' described in citefeaturespacesaturation. Finally we are able to show that saturation patterns converge early during training, allowing for a quicker cycle time during analysis