Automated Architecture Synthesis for Arbitrarily Structured Neural Networks

This paper offers a new perspective on Artificial Neural Networks (ANNs) architecture. Traditional ANNs commonly use tree-like or DAG structures for simplicity, which can be preset or determined by Neural Architecture Search (NAS). Yet, these structures restrict network collaboration and capability due to the absence of horizontal and backward communication. Biological neural systems, however, feature billions of neural units with highly complex connections, allowing each biological neuron to connect with others based on specific situations. Inspired by biological systems, we propose a novel framework that learns to construct arbitrary graph structures during training and introduce the concept of Neural Modules for organizing neural units, which facilitates communication between any nodes and collaboration among modules. Unlike traditional NAS methods that rely on DAG search spaces, our framework learns from complete graphs, enabling free communication between neurons akin to biological neural networks. Furthermore, we present a method to compute these structures and a regularization technique that organizes them into multiple independent, balanced neural modules. This approach reduces overfitting and improves efficiency through parallel computing. Overall, our method allows ANNs to learn effective arbitrary structures similar to biological ones. It is adaptable to various tasks and compatible across different scenarios, with experimental results demonstrating its potential.