# Quantifying and maximizing the information flux in recurrent neural networks

arXiv: https://arxiv.org/abs/2301.12892 · HTML: https://mlsanity.com/paper/2301.12892/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2023.

## Built on by (most cited first, 5 of 5)

- [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023)
- [Conceptual Cognitive Maps Formation with Neural Successor Networks and Word Embeddings](https://mlsanity.com/paper/2307.01577/) (2023)
- [Analysis of Argument Structure Constructions in a Deep Recurrent Language Model](https://mlsanity.com/paper/2408.03062/) (2024)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Author-Specific Linguistic Patterns Unveiled: A Deep Learning Study on Word Class Distributions](https://mlsanity.com/paper/2501.10072/) (2025)

## Abstract (arXiv)

Free-running Recurrent Neural Networks (RNNs), especially probabilistic models, generate an ongoing information flux that can be quantified with the mutual information Ileft[vecx(t),vecx(t!+!1)right] between subsequent system states vecx. Although, former studies have shown that I depends on the statistics of the network's connection weights, it is unclear (1) how to maximize I systematically and (2) how to quantify the flux in large systems where computing the mutual information becomes intractable. Here, we address these questions using Boltzmann machines as model systems. We find that in networks with moderately strong connections, the mutual information I is approximately a monotonic transformation of the root-mean-square averaged Pearson correlations between neuron-pairs, a quantity that can be efficiently computed even in large systems. Furthermore, evolutionary maximization of Ileft[vecx(t),vecx(t!+!1)right] reveals a general design principle for the weight matrices enabling the systematic construction of systems with a high spontaneous information flux. Finally, we simultaneously maximize information flux and the mean period length of cyclic attractors in the state space of these dynamical networks. Our results are potentially useful for the construction of RNNs that serve as short-time memories or pattern generators.