# Emergence of Network Motifs in Deep Neural Networks

arXiv: https://arxiv.org/abs/1912.12244 · HTML: https://mlsanity.com/paper/1912.12244/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2019.

## Built on by (most cited first, 9 of 9)

- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Structure and Performance of Fully Connected Neural Networks: Emerging Complex Network Properties](https://mlsanity.com/paper/2107.14062/) (2021)
- [Characterizing Learning Dynamics of Deep Neural Networks via Complex Networks](https://mlsanity.com/paper/2110.02628/) (2021)
- [Improving Deep Neural Network Random Initialization Through Neuronal Rewiring](https://mlsanity.com/paper/2207.08148/) (2022)
- [Deep Neural Networks as Complex Networks](https://mlsanity.com/paper/2209.05488/) (2022)
- [Deep Neural Networks via Complex Network Theory: a Perspective](https://mlsanity.com/paper/2404.11172/) (2024)
- [A developmental approach for training deep belief networks](https://mlsanity.com/paper/2207.05473/) (2022)
- [Graph-based methods coupled with specific distributional distances for adversarial attack detection](https://mlsanity.com/paper/2306.00042/) (2023)
- [Self-similarity Analysis in Deep Neural Networks](https://mlsanity.com/paper/2507.17785/) (2025)

## Abstract (arXiv)

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called "network motifs". In this article we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Our simulations show that the final network topology is primarily shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.