# Combating the Instability of Mutual Information-based Losses via Regularization

arXiv: https://arxiv.org/abs/2011.07932 · HTML: https://mlsanity.com/paper/2011.07932/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 5 of 5)

- [Understanding Probe Behaviors through Variational Bounds of Mutual Information](https://mlsanity.com/paper/2312.10019/) (2023)
- [TREET: TRansfer Entropy Estimation via Transformers](https://mlsanity.com/paper/2402.06919/) (2024)
- [Unsupervised Speech Representation Pooling Using Vector Quantization](https://mlsanity.com/paper/2304.03940/) (2023)
- [Benchmarking Mutual Information-based Loss Functions in Federated Learning](https://mlsanity.com/paper/2504.11877/) (2025)
- [A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds](https://mlsanity.com/paper/2510.05386/) (2025)

## Abstract (arXiv)

Notable progress has been made in numerous fields of machine learning based on neural network-driven mutual information (MI) bounds. However, utilizing the conventional MI-based losses is often challenging due to their practical and mathematical limitations. In this work, we first identify the symptoms behind their instability: (1) the neural network not converging even after the loss seemed to converge, and (2) saturating neural network outputs causing the loss to diverge. We mitigate both issues by adding a novel regularization term to the existing losses. We theoretically and experimentally demonstrate that added regularization stabilizes training. Finally, we present a novel benchmark that evaluates MI-based losses on both the MI estimation power and its capability on the downstream tasks, closely following the pre-existing supervised and contrastive learning settings. We evaluate six different MI-based losses and their regularized counterparts on multiple benchmarks to show that our approach is simple yet effective.