# Understanding Deep Learning Generalization by Maximum Entropy

arXiv: https://arxiv.org/abs/1711.07758 · HTML: https://mlsanity.com/paper/1711.07758/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2017.

## Built on by (most cited first, 3 of 3)

- [LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization](https://mlsanity.com/paper/2410.17020/) (2024)
- [High-entropy Advantage in Neural Networks' Generalizability](https://mlsanity.com/paper/2503.13145/) (2025)
- [Cellular Development Follows the Path of Minimum Action](https://mlsanity.com/paper/2504.08096/) (2025)

## Abstract (arXiv)

Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development.