# Understanding CNN Fragility When Learning With Imbalanced Data

arXiv: https://arxiv.org/abs/2210.09465 · HTML: https://mlsanity.com/paper/2210.09465/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [Deep Imbalanced Learning for Face Recognition and Attribute Prediction](https://mlsanity.com/paper/1806.00194/) (2018)
- [Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning](https://mlsanity.com/paper/2001.01385/) (2020)
- [Adjusting Decision Boundary for Class Imbalanced Learning](https://mlsanity.com/paper/1912.01857/) (2019)
- [Characterizing the Decision Boundary of Deep Neural Networks](https://mlsanity.com/paper/1912.11460/) (2019)
- [Deep nets for local manifold learning](https://mlsanity.com/paper/1607.07110/) (2016)
- [Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power](https://mlsanity.com/paper/2205.13863/) (2022)
- [Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study](https://mlsanity.com/paper/2002.01810/) (2020)
- [ReMix: Calibrated Resampling for Class Imbalance in Deep learning](https://mlsanity.com/paper/2012.02312/) (2020)
- [Efficient Augmentation for Imbalanced Deep Learning](https://mlsanity.com/paper/2207.06080/) (2022)

## Built on by (most cited first, 10 of 10)

- [DT-DDNN: A Physical Layer Security Attack Detector in 5G RF Domain for CAVs](https://mlsanity.com/paper/2403.02645/) (2024)
- [Towards Understanding How Data Augmentation Works with Imbalanced Data](https://mlsanity.com/paper/2304.05895/) (2023)
- [Synthetic Data Augmentation for Medical Audio Classification: A Preliminary Evaluation](https://mlsanity.com/paper/2602.02955/) (2026)
- [Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing](https://mlsanity.com/paper/2606.01221/) (2026)
- [Interpretable ML for Imbalanced Data](https://mlsanity.com/paper/2212.07743/) (2022)
- [Dual-path convolutional neural network using micro-FTIR imaging to predict breast cancer subtypes and biomarkers levels: estrogen receptor, progesterone receptor, HER2 and Ki67](https://mlsanity.com/paper/2310.15099/) (2023)
- [Learning Confidence Bounds for Classification with Imbalanced Data](https://mlsanity.com/paper/2407.11878/) (2024)
- [Conformal-in-the-Loop for Learning with Imbalanced Noisy Data](https://mlsanity.com/paper/2411.02281/) (2024)
- [Quantization-Aware Neuromorphic Architecture for Skin Lesion Classification on Resource-Constrained Devices](https://mlsanity.com/paper/2507.15958/) (2025)
- [DADIR: Density-Aware Data-level Imbalanced Regression Framework](https://mlsanity.com/paper/2607.17178/) (2026)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have achieved impressive results on imbalanced image data, but they still have difficulty generalizing to minority classes and their decisions are difficult to interpret. These problems are related because the method by which CNNs generalize to minority classes, which requires improvement, is wrapped in a blackbox. To demystify CNN decisions on imbalanced data, we focus on their latent features. Although CNNs embed the pattern knowledge learned from a training set in model parameters, the effect of this knowledge is contained in feature and classification embeddings (FE and CE). These embeddings can be extracted from a trained model and their global, class properties (e.g., frequency, magnitude and identity) can be analyzed. We find that important information regarding the ability of a neural network to generalize to minority classes resides in the class top-K CE and FE. We show that a CNN learns a limited number of class top-K CE per category, and that their number and magnitudes vary based on whether the same class is balanced or imbalanced. This calls into question whether a CNN has learned intrinsic class features, or merely frequently occurring ones that happen to exist in the sampled class distribution. We also hypothesize that latent class diversity is as important as the number of class examples, which has important implications for re-sampling and cost-sensitive methods. These methods generally focus on rebalancing model weights, class numbers and margins; instead of diversifying class latent features through augmentation. We also demonstrate that a CNN has difficulty generalizing to test data if the magnitude of its top-K latent features do not match the training set. We use three popular image datasets and two cost-sensitive algorithms commonly employed in imbalanced learning for our experiments.