# Automated Architecture Synthesis for Arbitrarily Structured Neural Networks

arXiv: https://arxiv.org/abs/2306.02157 · HTML: https://mlsanity.com/paper/2306.02157/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 3 Jun 2023.

## Builds on (most cited first, 4 of 4)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [MaskConnect: Connectivity Learning by Gradient Descent](https://mlsanity.com/paper/1807.11473/) (2018)

## Abstract (arXiv)

This paper offers a new perspective on Artificial Neural Networks (ANNs) architecture. Traditional ANNs commonly use tree-like or DAG structures for simplicity, which can be preset or determined by Neural Architecture Search (NAS). Yet, these structures restrict network collaboration and capability due to the absence of horizontal and backward communication. Biological neural systems, however, feature billions of neural units with highly complex connections, allowing each biological neuron to connect with others based on specific situations. Inspired by biological systems, we propose a novel framework that learns to construct arbitrary graph structures during training and introduce the concept of Neural Modules for organizing neural units, which facilitates communication between any nodes and collaboration among modules. Unlike traditional NAS methods that rely on DAG search spaces, our framework learns from complete graphs, enabling free communication between neurons akin to biological neural networks. Furthermore, we present a method to compute these structures and a regularization technique that organizes them into multiple independent, balanced neural modules. This approach reduces overfitting and improves efficiency through parallel computing. Overall, our method allows ANNs to learn effective arbitrary structures similar to biological ones. It is adaptable to various tasks and compatible across different scenarios, with experimental results demonstrating its potential.