# Leveraging the Graph Structure of Neural Network Training Dynamics

arXiv: https://arxiv.org/abs/2111.05410 · HTML: https://mlsanity.com/paper/2111.05410/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 9 Nov 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Predicting Neural Network Accuracy from Weights](https://mlsanity.com/paper/2002.11448/) (2020)
- [Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology](https://mlsanity.com/paper/1812.09764/) (2018)
- [Graph Structure of Neural Networks](https://mlsanity.com/paper/2007.06559/) (2020)
- [Clusterability in Neural Networks](https://mlsanity.com/paper/2103.03386/) (2021)
- [Modern temporal network theory: A colloquium](https://mlsanity.com/paper/1508.01303/) (2015)
- [Graph Summarization Methods and Applications: A Survey](https://mlsanity.com/paper/1612.04883/) (2016)
- [Mining Persistent Activity in Continually Evolving Networks](https://mlsanity.com/paper/2006.15410/) (2020)
- [Deep learning in bioinformatics: introduction, application, and perspective in big data era](https://mlsanity.com/paper/1903.00342/) (2019)

## Built on by (most cited first, 1 of 1)

- [Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations](https://mlsanity.com/paper/2305.16886/) (2023)

## Abstract (arXiv)

Understanding the training dynamics of deep neural networks (DNNs) is important as it can lead to improved training efficiency and task performance. Recent works have demonstrated that representing the wirings of static graph cannot capture how DNNs change over the course of training. Thus, in this work, we propose a compact, expressive temporal graph framework that effectively captures the dynamics of many workhorse architectures in computer vision. Specifically, it extracts an informative summary of graph properties (e.g., eigenvector centrality) over a sequence of DNN graphs obtained during training. We demonstrate that our framework captures useful dynamics by accurately predicting trained, task performance when using a summary over early training epochs (<5) across four different architectures and two image datasets. Moreover, by using a novel, highly-scalable DNN graph representation, we also show that the proposed framework captures generalizable dynamics as summaries extracted from smaller-width networks are effective when evaluated on larger widths.