# Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

arXiv: https://arxiv.org/abs/2006.15766 · HTML: https://mlsanity.com/paper/2006.15766/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 30 of 49)

- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [FINE Samples for Learning with Noisy Labels](https://mlsanity.com/paper/2102.11628/) (2021)
- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [Robust Learning with Progressive Data Expansion Against Spurious Correlation](https://mlsanity.com/paper/2306.04949/) (2023)
- [Correlated Input-Dependent Label Noise in Large-Scale Image Classification](https://mlsanity.com/paper/2105.10305/) (2021)
- [Robust Long-Tailed Learning under Label Noise](https://mlsanity.com/paper/2108.11569/) (2021)
- [SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning](https://mlsanity.com/paper/2209.10365/) (2022)
- [CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning](https://mlsanity.com/paper/2202.05613/) (2022)
- [When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method](https://mlsanity.com/paper/2211.10955/) (2022)
- [Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection](https://mlsanity.com/paper/2111.14932/) (2021)
- [Identifying Hard Noise in Long-Tailed Sample Distribution](https://mlsanity.com/paper/2207.13378/) (2022)
- [Label-Noise Learning with Intrinsically Long-Tailed Data](https://mlsanity.com/paper/2208.09833/) (2022)
- [Late Stopping: Avoiding Confidently Learning from Mislabeled Examples](https://mlsanity.com/paper/2308.13862/) (2023)
- [Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data](https://mlsanity.com/paper/2201.00849/) (2021)
- [Fairness through Aleatoric Uncertainty](https://mlsanity.com/paper/2304.03646/) (2023)
- [Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints](https://mlsanity.com/paper/2211.01052/) (2022)
- [Wrapped Cauchy Distributed Angular Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2305.18732/) (2023)
- [A Survey of Deep Long-Tail Classification Advancements](https://mlsanity.com/paper/2404.15593/) (2024)
- [Prototypical Classifier for Robust Class-Imbalanced Learning](https://mlsanity.com/paper/2110.11553/) (2021)
- [Fine tuning Pre trained Models for Robustness Under Noisy Labels](https://mlsanity.com/paper/2310.17668/) (2023)
- [Dynamic Loss For Robust Learning](https://mlsanity.com/paper/2211.12506/) (2022)
- [Bias Amplification Enhances Minority Group Performance](https://mlsanity.com/paper/2309.06717/) (2023)
- [Neural Active Learning on Heteroskedastic Distributions](https://mlsanity.com/paper/2211.00928/) (2022)
- [Regularization Penalty Optimization for Addressing Data Quality Variance in OoD Algorithms](https://mlsanity.com/paper/2206.05749/) (2022)

## Abstract (arXiv)

Real-world large-scale datasets are heteroskedastic and imbalanced -- labels have varying levels of uncertainty and label distributions are long-tailed. Heteroskedasticity and imbalance challenge deep learning algorithms due to the difficulty of distinguishing among mislabeled, ambiguous, and rare examples. Addressing heteroskedasticity and imbalance simultaneously is under-explored. We propose a data-dependent regularization technique for heteroskedastic datasets that regularizes different regions of the input space differently. Inspired by the theoretical derivation of the optimal regularization strength in a one-dimensional nonparametric classification setting, our approach adaptively regularizes the data points in higher-uncertainty, lower-density regions more heavily. We test our method on several benchmark tasks, including a real-world heteroskedastic and imbalanced dataset, WebVision. Our experiments corroborate our theory and demonstrate a significant improvement over other methods in noise-robust deep learning.