# Structurally Flexible Neural Networks: Evolving the Building Blocks for General Agents

arXiv: https://arxiv.org/abs/2404.15193 · HTML: https://mlsanity.com/paper/2404.15193/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 6 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 20 of 20)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/2109.02869/) (2021)
- [Meta Learning Backpropagation And Improving It](https://mlsanity.com/paper/2012.14905/) (2020)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [Introducing Symmetries to Black Box Meta Reinforcement Learning](https://mlsanity.com/paper/2109.10781/) (2021)
- [Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs](https://mlsanity.com/paper/2307.08197/) (2023)
- [Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules](https://mlsanity.com/paper/2104.07959/) (2021)
- [Minimal Neural Network Models for Permutation Invariant Agents](https://mlsanity.com/paper/2205.07868/) (2022)
- [Learning to Act through Evolution of Neural Diversity in Random Neural Networks](https://mlsanity.com/paper/2305.15945/) (2023)

## Built on by (most cited first, 4 of 4)

- [Continuous Thought Machines](https://mlsanity.com/paper/2505.05522/) (2025)
- [Evolving Self-Assembling Neural Networks: From Spontaneous Activity to Experience-Dependent Learning](https://mlsanity.com/paper/2406.09787/) (2024)
- [The Generalist Brain Module: Module Repetition in Neural Networks in Light of the Minicolumn Hypothesis](https://mlsanity.com/paper/2507.12473/) (2025)
- [Distilling a Modular Reservoir Through a Genomic Bottleneck](https://mlsanity.com/paper/2606.28380/) (2026)

## Abstract (arXiv)

Artificial neural networks used for reinforcement learning are structurally rigid, meaning that each optimized parameter of the network is tied to its specific placement in the network structure. It also means that a network only works with pre-defined and fixed input- and output sizes. This is a consequence of having the number of optimized parameters being directly dependent on the structure of the network. Structural rigidity limits the ability to optimize parameters of policies across multiple environments that do not share input and output spaces. Here, we evolve a set of neurons and plastic synapses each represented by a gated recurrent unit (GRU). During optimization, the parameters of these fundamental units of a neural network are optimized in different random structural configurations. Earlier work has shown that parameter sharing between units is important for making structurally flexible neurons We show that it is possible to optimize a set of distinct neuron- and synapse types allowing for a mitigation of the symmetry dilemma. We demonstrate this by optimizing a single set of neurons and synapses to solve multiple reinforcement learning control tasks simultaneously.