# Information-Theoretic Perspectives on Optimizers

arXiv: https://arxiv.org/abs/2502.20763 · HTML: https://mlsanity.com/paper/2502.20763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 8 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum](https://mlsanity.com/paper/2006.15815/) (2020)
- [Information Theory with Kernel Methods](https://mlsanity.com/paper/2202.08545/) (2022)
- [DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies](https://mlsanity.com/paper/2301.08164/) (2023)
- [Matrix Information Theory for Self-Supervised Learning](https://mlsanity.com/paper/2305.17326/) (2023)
- [Information Flow in Self-Supervised Learning](https://mlsanity.com/paper/2309.17281/) (2023)
- [RelationMatch: Matching In-batch Relationships for Semi-supervised Learning](https://mlsanity.com/paper/2305.10397/) (2023)

## Abstract (arXiv)

The interplay of optimizers and architectures in neural networks is complicated and hard to understand why some optimizers work better on some specific architectures. In this paper, we find that the traditionally used sharpness metric does not fully explain the intricate interplay and introduces information-theoretic metrics called entropy gap to better help analyze. It is found that both sharpness and entropy gap affect the performance, including the optimization dynamic and generalization. We further use information-theoretic tools to understand a recently proposed optimizer called Lion and find ways to improve it.