# Transfer entropy-based feedback improves performance in artificial neural networks

arXiv: https://arxiv.org/abs/1706.04265 · HTML: https://mlsanity.com/paper/1706.04265/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2017.

## Built on by (most cited first, 5 of 5)

- [Learning in Feedforward Neural Networks Accelerated by Transfer Entropy](https://mlsanity.com/paper/2104.14616/) (2021)
- [Applications of Entropy in Data Analysis and Machine Learning: A Review](https://mlsanity.com/paper/2503.02921/) (2025)
- [Learning in Convolutional Neural Networks Accelerated by Transfer Entropy](https://mlsanity.com/paper/2404.02943/) (2024)
- [Information Plane Analysis Visualization in Deep Learning via Transfer Entropy](https://mlsanity.com/paper/2404.01364/) (2024)
- [Transfer Entropy in Graph Convolutional Neural Networks](https://mlsanity.com/paper/2406.06632/) (2024)

## Abstract (arXiv)

The structure of the majority of modern deep neural networks is characterized by uni- directional feed-forward connectivity across a very large number of layers. By contrast, the architecture of the cortex of vertebrates contains fewer hierarchical levels but many recurrent and feedback connections. Here we show that a small, few-layer artificial neural network that employs feedback will reach top level performance on a standard benchmark task, otherwise only obtained by large feed-forward structures. To achieve this we use feed-forward transfer entropy between neurons to structure feedback connectivity. Transfer entropy can here intuitively be understood as a measure for the relevance of certain pathways in the network, which are then amplified by feedback. Feedback may therefore be key for high network performance in small brain-like architectures.