# Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition

arXiv: https://arxiv.org/abs/2510.27651 · HTML: https://mlsanity.com/paper/2510.27651/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 53 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [The HSIC Bottleneck: Deep Learning without Back-Propagation](https://mlsanity.com/paper/1908.01580/) (2019)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [Understanding Synthetic Gradients and Decoupled Neural Interfaces](https://mlsanity.com/paper/1703.00522/) (2017)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [Understanding Autoencoders with Information Theoretic Concepts](https://mlsanity.com/paper/1804.00057/) (2018)
- [Learning to solve the credit assignment problem](https://mlsanity.com/paper/1906.00889/) (2019)
- [Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks](https://mlsanity.com/paper/2006.07123/) (2020)
- [Scalable Mutual Information Estimation using Dependence Graphs](https://mlsanity.com/paper/1801.09125/) (2018)
- [Flatness is a False Friend](https://mlsanity.com/paper/2006.09091/) (2020)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [On Kernel Method-Based Connectionist Models and Supervised Deep Learning Without Backpropagation](https://mlsanity.com/paper/1802.03774/) (2018)

## Abstract (arXiv)

Modern deep neural networks (DNNs) are typically trained with a global cross-entropy loss in a supervised end-to-end manner: neurons need to store their outgoing weights; training alternates between a forward pass (computation) and a top-down backward pass (learning) which is biologically implausible. Alternatively, greedy layer-wise training eliminates the need for cross-entropy loss and backpropagation. By avoiding the computation of intermediate gradients and the storage of intermediate outputs, it reduces memory usage and helps mitigate issues such as vanishing or exploding gradients. However, most existing layer-wise training approaches have been evaluated only on relatively small datasets with simple deep architectures. In this paper, we first systematically analyze the training dynamics of popular convolutional neural networks (CNNs) trained by stochastic gradient descent (SGD) through an information-theoretic lens. Our findings reveal that networks converge layer-by-layer from bottom to top and that the flow of information adheres to a Markov information bottleneck principle. Building on these observations, we propose a novel layer-wise training approach based on the recently developed deterministic information bottleneck (DIB) and the matrix-based R'enyi's α-order entropy functional. Specifically, each layer is trained jointly with an auxiliary classifier that connects directly to the output layer, enabling the learning of minimal sufficient task-relevant representations. We empirically validate the effectiveness of our training procedure on CIFAR-10 and CIFAR-100 using modern deep CNNs and further demonstrate its applicability to a practical task involving traffic sign recognition. Our approach not only outperforms existing layer-wise training baselines but also achieves performance comparable to SGD.