# Graph Structure of Neural Networks

arXiv: https://arxiv.org/abs/2007.06559 · HTML: https://mlsanity.com/paper/2007.06559/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 30 of 68)

- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [Identity-aware Graph Neural Networks](https://mlsanity.com/paper/2101.10320/) (2021)
- [Graph Neural Networks in Network Neuroscience](https://mlsanity.com/paper/2106.03535/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)
- [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023)
- [Automated Machine Learning on Graphs: A Survey](https://mlsanity.com/paper/2103.00742/) (2021)
- [Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks](https://mlsanity.com/paper/2205.10664/) (2022)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2406.10685/) (2024)
- [The Elastic Lottery Ticket Hypothesis](https://mlsanity.com/paper/2103.16547/) (2021)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective](https://mlsanity.com/paper/2112.14531/) (2021)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Exploring Robust Architectures for Deep Artificial Neural Networks](https://mlsanity.com/paper/2106.15850/) (2021)
- [Quantifying Local Specialization in Deep Neural Networks](https://mlsanity.com/paper/2110.08058/) (2021)
- [Automated Graph Machine Learning: Approaches, Libraries, Benchmarks and Directions](https://mlsanity.com/paper/2201.01288/) (2022)
- [Building Extraction from Remote Sensing Images via an Uncertainty-Aware Network](https://mlsanity.com/paper/2307.12309/) (2023)
- [Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction](https://mlsanity.com/paper/2406.04934/) (2024)
- [Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics](https://mlsanity.com/paper/2108.11939/) (2021)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [A Unified and Biologically-Plausible Relational Graph Representation of Vision Transformers](https://mlsanity.com/paper/2206.11073/) (2022)
- [Improving Deep Neural Network Random Initialization Through Neuronal Rewiring](https://mlsanity.com/paper/2207.08148/) (2022)
- [Core-Periphery Principle Guided Redesign of Self-Attention in Transformers](https://mlsanity.com/paper/2303.15569/) (2023)
- [NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning](https://mlsanity.com/paper/2306.10792/) (2023)
- [Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks](https://mlsanity.com/paper/2310.19470/) (2023)
- [Graph-based Neural Architecture Search with Operation Embeddings](https://mlsanity.com/paper/2105.04885/) (2021)
- [GraphPNAS: Learning Distribution of Good Neural Architectures via Deep Graph Generative Models](https://mlsanity.com/paper/2211.15155/) (2022)
- [Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis](https://mlsanity.com/paper/2205.05662/) (2022)

## Abstract (arXiv)

Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that: (1) a "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural network's performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.