# Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data

arXiv: https://arxiv.org/abs/2212.13827 · HTML: https://mlsanity.com/paper/2212.13827/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 65 references that are in the map.
- First posted on arXiv 28 Dec 2022.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [Parametric Contrastive Learning](https://mlsanity.com/paper/2107.12028/) (2021)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [Improving Calibration for Long-Tailed Recognition](https://mlsanity.com/paper/2104.00466/) (2021)
- [Distribution Alignment: A Unified Framework for Long-tail Visual Recognition](https://mlsanity.com/paper/2103.16370/) (2021)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)

## Built on by (most cited first, 28 of 28)

- [ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition](https://mlsanity.com/paper/2308.07815/) (2023)
- [A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization](https://mlsanity.com/paper/2310.04752/) (2023)
- [An SDE for Modeling SAM: Theory and Insights](https://mlsanity.com/paper/2301.08203/) (2023)
- [DirMixE: Harnessing Test Agnostic Long-tail Recognition with Hierarchical Label Variations](https://mlsanity.com/paper/2405.07780/) (2024)
- [Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2410.21042/) (2024)
- [DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets](https://mlsanity.com/paper/2404.02900/) (2024)
- [Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification](https://mlsanity.com/paper/2505.01660/) (2025)
- [Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition](https://mlsanity.com/paper/2410.06109/) (2024)
- [Long-tailed Recognition with Model Rebalancing](https://mlsanity.com/paper/2510.08177/) (2025)
- [Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning](https://mlsanity.com/paper/2405.20439/) (2024)
- [Curvature Learning for Generalization of Hyperbolic Neural Networks](https://mlsanity.com/paper/2508.17232/) (2025)
- [DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognition](https://mlsanity.com/paper/2403.05170/) (2024)
- [FairDD: Fair Dataset Distillation](https://mlsanity.com/paper/2411.19623/) (2024)
- [SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAM](https://mlsanity.com/paper/2412.13715/) (2024)
- [Prior2Posterior: Model Prior Correction for Long-Tailed Learning](https://mlsanity.com/paper/2412.16540/) (2024)
- [Biased Binary Attribute Classifiers Ignore the Majority Classes](https://mlsanity.com/paper/2403.14435/) (2024)
- [Taming the Tail: Leveraging Asymmetric Loss and Pade Approximation to Overcome Medical Image Long-Tailed Class Imbalance](https://mlsanity.com/paper/2410.04084/) (2024)
- [FEDTAIL: Federated Long-Tailed Domain Generalization with Sharpness-Guided Gradient Matching](https://mlsanity.com/paper/2506.08518/) (2025)
- [Minimizing Layerwise Activation Norm Improves Generalization in Federated Learning](https://mlsanity.com/paper/2512.08314/) (2025)
- [Reframing Long-Tailed Learning via Loss Landscape Geometry](https://mlsanity.com/paper/2603.21217/) (2026)
- [Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions](https://mlsanity.com/paper/2402.05400/) (2024)
- [Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model](https://mlsanity.com/paper/2404.00285/) (2024)
- [Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification](https://mlsanity.com/paper/2410.03588/) (2024)
- [Deep Minimax Classifiers for Imbalanced Datasets with a Small Number of Minority Samples](https://mlsanity.com/paper/2502.16948/) (2025)
- [NCSAM Noise-Compensated Sharpness-Aware Minimization for Noisy Label Learning](https://mlsanity.com/paper/2601.19947/) (2026)
- [Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation](https://mlsanity.com/paper/2603.10048/) (2026)
- [Why Not Hyperparameter-Friendly Optimisation? A Monotonic Adaptive Norm Rescaling Approach For Long-Tailed Recognition](https://mlsanity.com/paper/2606.02526/) (2026)
- [Federated Learning for Feature Generalization with Convex Constraints](https://mlsanity.com/paper/2606.14416/) (2026)

## Abstract (arXiv)

Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used to enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin-based techniques. Specifically, we examine the spectral density of Hessian of class-wise loss, through which we observe that the network weights converge to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes. We further theoretically and empirically demonstrate that Sharpness-Aware Minimization (SAM), a recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results in a 6.2% increase in accuracy on the minority classes over the state-of-the-art Vector Scaling Loss, leading to an overall average increase of 4% across imbalanced datasets. The code is available at: https://github.com/val-iisc/Saddle-LongTail.