# Learning in Convolutional Neural Networks Accelerated by Transfer Entropy

arXiv: https://arxiv.org/abs/2404.02943 · HTML: https://mlsanity.com/paper/2404.02943/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 47 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [Transfer entropy-based feedback improves performance in artificial neural networks](https://mlsanity.com/paper/1706.04265/) (2017)
- [Learning in Feedforward Neural Networks Accelerated by Transfer Entropy](https://mlsanity.com/paper/2104.14616/) (2021)
- [Semiotic Aggregation in Deep Learning](https://mlsanity.com/paper/2104.10931/) (2021)
- [Measures of Causality in Complex Datasets with application to financial data](https://mlsanity.com/paper/1401.1457/) (2014)

## Built on by (most cited first, 2 of 2)

- [Applications of Entropy in Data Analysis and Machine Learning: A Review](https://mlsanity.com/paper/2503.02921/) (2025)
- [Transfer Entropy in Graph Convolutional Neural Networks](https://mlsanity.com/paper/2406.06632/) (2024)

## Abstract (arXiv)

Recently, there is a growing interest in applying Transfer Entropy (TE) in quantifying the effective connectivity between artificial neurons. In a feedforward network, the TE can be used to quantify the relationships between neuron output pairs located in different layers. Our focus is on how to include the TE in the learning mechanisms of a Convolutional Neural Network (CNN) architecture. We introduce a novel training mechanism for CNN architectures which integrates the TE feedback connections. Adding the TE feedback parameter accelerates the training process, as fewer epochs are needed. On the flip side, it adds computational overhead to each epoch. According to our experiments on CNN classifiers, to achieve a reasonable computational overhead--accuracy trade-off, it is efficient to consider only the inter-neural information transfer of a random subset of the neuron pairs from the last two fully connected layers. The TE acts as a smoothing factor, generating stability and becoming active only periodically, not after processing each input sample. Therefore, we can consider the TE is in our model a slowly changing meta-parameter.