# Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

arXiv: https://arxiv.org/abs/2201.04194 · HTML: https://mlsanity.com/paper/2201.04194/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 51 references that are in the map.
- First posted on arXiv 11 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019)
- [NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size](https://mlsanity.com/paper/2009.00437/) (2020)
- [HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark](https://mlsanity.com/paper/2103.10584/) (2021)
- [Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization](https://mlsanity.com/paper/1908.09375/) (2019)
- [NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search](https://mlsanity.com/paper/2001.10422/) (2020)
- [An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis](https://mlsanity.com/paper/1703.00560/) (2017)
- [Inferring Degrees from Incomplete Networks and Nonlinear Dynamics](https://mlsanity.com/paper/2004.10546/) (2020)
- [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020)
- [Learning to Rank Learning Curves](https://mlsanity.com/paper/2006.03361/) (2020)
- [Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition](https://mlsanity.com/paper/2006.14371/) (2020)
- [Dynamical Systems and Neural Networks](https://mlsanity.com/paper/2004.11826/) (2020)
- [True Nonlinear Dynamics from Incomplete Networks](https://mlsanity.com/paper/2001.06722/) (2020)

## Built on by (most cited first, 3 of 3)

- [Learning to Rank Normalized Entropy Curves with Differentiable Window Transformation](https://mlsanity.com/paper/2301.10443/) (2023)
- [Artificial Bee Colony optimization of Deep Convolutional Neural Networks in the context of Biomedical Imaging](https://mlsanity.com/paper/2402.15246/) (2024)
- [Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry](https://mlsanity.com/paper/2609.02373/) (2026)

## Abstract (arXiv)

Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping φ, converting a neural network GA to a directed line graph GB that is defined on those edges in GA. Next, we derive a neural capacitance metric βrm eff as a predictive measure universally capturing the generalization capability of GA on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets, including CIFAR10, CIFAR100, SVHN, Fashion MNIST and Birds, to evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.