# Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator

arXiv: https://arxiv.org/abs/1911.10688 · HTML: https://mlsanity.com/paper/1911.10688/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2019.

## Built on by (most cited first, 25 of 25)

- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity](https://mlsanity.com/paper/2102.03065/) (2021)
- [FeatUp: A Model-Agnostic Framework for Features at Any Resolution](https://mlsanity.com/paper/2403.10516/) (2024)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Frustratingly Easy Transferability Estimation](https://mlsanity.com/paper/2106.09362/) (2021)
- [Learning Bounds for Open-Set Learning](https://mlsanity.com/paper/2106.15792/) (2021)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learning](https://mlsanity.com/paper/2109.03560/) (2021)
- [Generalized Data Weighting via Class-level Gradient Manipulation](https://mlsanity.com/paper/2111.00056/) (2021)
- [POODLE: Improving Few-shot Learning via Penalizing Out-of-Distribution Samples](https://mlsanity.com/paper/2206.04679/) (2022)
- [NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs](https://mlsanity.com/paper/2208.10010/) (2022)
- [What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs](https://mlsanity.com/paper/2206.09358/) (2022)
- [From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios](https://mlsanity.com/paper/2403.07403/) (2024)
- [MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes](https://mlsanity.com/paper/2404.09232/) (2024)
- [Box-based Refinement for Weakly Supervised and Unsupervised Localization Tasks](https://mlsanity.com/paper/2309.03874/) (2023)
- [Informative Class Activation Maps](https://mlsanity.com/paper/2106.10472/) (2021)
- [Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference](https://mlsanity.com/paper/2411.14035/) (2024)
- [Learning a Weight Map for Weakly-Supervised Localization](https://mlsanity.com/paper/2111.14131/) (2021)
- [Text recognition on images using pre-trained CNN](https://mlsanity.com/paper/2302.05105/) (2023)
- [Improving Mass Detection in Mammography Images: A Study of Weakly Supervised Learning and Class Activation Map Methods](https://mlsanity.com/paper/2308.03486/) (2023)
- [Position-Sensing Graph Neural Networks: Proactively Learning Nodes Relative Positions](https://mlsanity.com/paper/2105.11346/) (2021)
- [Learning Consistency from High-quality Pseudo-labels for Weakly Supervised Object Localization](https://mlsanity.com/paper/2203.09803/) (2022)
- [Explored An Effective Methodology for Fine-Grained Snake Recognition](https://mlsanity.com/paper/2207.11637/) (2022)
- [Mixup Augmentation with Multiple Interpolations](https://mlsanity.com/paper/2406.01417/) (2024)
- [Remedying Coarsening-Based GNN Training under Heterophily via Adaptive Complementary Enhancement](https://mlsanity.com/paper/2607.21885/) (2026)

## Abstract (arXiv)

Mutual information is widely applied to learn latent representations of observations, whilst its implication in classification neural networks remain to be better explained. We show that optimising the parameters of classification neural networks with softmax cross-entropy is equivalent to maximising the mutual information between inputs and labels under the balanced data assumption. Through experiments on synthetic and real datasets, we show that softmax cross-entropy can estimate mutual information approximately. When applied to image classification, this relation helps approximate the point-wise mutual information between an input image and a label without modifying the network structure. To this end, we propose infoCAM, informative class activation map, which highlights regions of the input image that are the most relevant to a given label based on differences in information. The activation map helps localise the target object in an input image. Through experiments on the semi-supervised object localisation task with two real-world datasets, we evaluate the effectiveness of our information-theoretic approach.