# Conditional Mutual Information Constrained Deep Learning for Classification

arXiv: https://arxiv.org/abs/2309.09123 · HTML: https://mlsanity.com/paper/2309.09123/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra](https://mlsanity.com/paper/2008.11865/) (2020)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Rethinking Feature Distribution for Loss Functions in Image Classification](https://mlsanity.com/paper/1803.02988/) (2018)
- [Separation and Concentration in Deep Networks](https://mlsanity.com/paper/2012.10424/) (2020)
- [Contrastive-center loss for deep neural networks](https://mlsanity.com/paper/1707.07391/) (2017)
- [Building a Regular Decision Boundary with Deep Networks](https://mlsanity.com/paper/1703.01775/) (2017)

## Built on by (most cited first, 18 of 18)

- [Knowledge Distillation Based on Transformed Teacher Matching](https://mlsanity.com/paper/2402.11148/) (2024)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation](https://mlsanity.com/paper/2506.17307/) (2025)
- [Distribution Alignment for Fully Test-Time Adaptation with Dynamic Online Data Streams](https://mlsanity.com/paper/2407.12128/) (2024)
- [Information-Guided Diffusion Sampling for Dataset Distillation](https://mlsanity.com/paper/2507.04619/) (2025)
- [ASMIL: Attention-Stabilized Multiple Instance Learning for Whole Slide Imaging](https://mlsanity.com/paper/2603.06658/) (2026)
- [Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information](https://mlsanity.com/paper/2412.09945/) (2024)
- [Towards Undistillable Models by Minimizing Conditional Mutual Information](https://mlsanity.com/paper/2507.00012/) (2025)
- [Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers](https://mlsanity.com/paper/2601.02543/) (2026)
- [Leveraging Conditional Mutual Information to Improve Large Language Model Fine-Tuning For Classification](https://mlsanity.com/paper/2502.11258/) (2025)
- [Reducing Class-Wise Performance Disparity via Margin Regularization](https://mlsanity.com/paper/2602.00205/) (2026)
- [JPEG Compliant Compression for Both Human and Machine, A Report](https://mlsanity.com/paper/2503.10912/) (2025)
- [Improving Open-Set Semantic Segmentation in 3D Point Clouds by Conditional Channel Capacity Maximization: Preliminary Results](https://mlsanity.com/paper/2505.11521/) (2025)
- [InfoSculpt: Sculpting the Latent Space for Generalized Category Discovery](https://mlsanity.com/paper/2601.10098/) (2026)
- [Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing](https://mlsanity.com/paper/2601.16812/) (2026)
- [Beyond Illumination: A Conditional Mutual Information-Guided Network for Low-Light Image Enhancement](https://mlsanity.com/paper/2608.01886/) (2026)
- [Temperature-Adaptive Transformed Teacher Matching](https://mlsanity.com/paper/2608.29099/) (2026)
- [CARA: Collision-Aware Resolution Adaptation for Multiresolution Hash Encoding Based Image Fitting](https://mlsanity.com/paper/2609.18554/) (2026)

## Abstract (arXiv)

The concepts of conditional mutual information (CMI) and normalized conditional mutual information (NCMI) are introduced to measure the concentration and separation performance of a classification deep neural network (DNN) in the output probability distribution space of the DNN, where CMI and the ratio between CMI and NCMI represent the intra-class concentration and inter-class separation of the DNN, respectively. By using NCMI to evaluate popular DNNs pretrained over ImageNet in the literature, it is shown that their validation accuracies over ImageNet validation data set are more or less inversely proportional to their NCMI values. Based on this observation, the standard deep learning (DL) framework is further modified to minimize the standard cross entropy function subject to an NCMI constraint, yielding CMI constrained deep learning (CMIC-DL). A novel alternating learning algorithm is proposed to solve such a constrained optimization problem. Extensive experiment results show that DNNs trained within CMIC-DL outperform the state-of-the-art models trained within the standard DL and other loss functions in the literature in terms of both accuracy and robustness against adversarial attacks. In addition, visualizing the evolution of learning process through the lens of CMI and NCMI is also advocated.