# Distributional Robustness Loss for Long-tail Learning

arXiv: https://arxiv.org/abs/2104.03066 · HTML: https://mlsanity.com/paper/2104.03066/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Feature Space Augmentation for Long-Tailed Data](https://mlsanity.com/paper/2008.03673/) (2020)
- [M2m: Imbalanced Classification via Major-to-minor Translation](https://mlsanity.com/paper/2004.00431/) (2020)
- [Distributionally Robust Logistic Regression](https://mlsanity.com/paper/1509.09259/) (2015)
- [On the Robustness of Semantic Segmentation Models to Adversarial Attacks](https://mlsanity.com/paper/1711.09856/) (2017)
- [Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspective](https://mlsanity.com/paper/2002.10826/) (2020)
- [Dynamic Curriculum Learning for Imbalanced Data Classification](https://mlsanity.com/paper/1901.06783/) (2019)
- [Data-Driven Robust Optimization](https://mlsanity.com/paper/1401.0212/) (2013)
- [Striking the Right Balance with Uncertainty](https://mlsanity.com/paper/1901.07590/) (2019)
- [Learning to Segment the Tail](https://mlsanity.com/paper/2004.00900/) (2020)
- [Factors in Finetuning Deep Model for object detection](https://mlsanity.com/paper/1601.05150/) (2016)
- [Synthetic Examples Improve Generalization for Rare Classes](https://mlsanity.com/paper/1904.05916/) (2019)
- [Unravelling Robustness of Deep Learning based Face Recognition Against Adversarial Attacks](https://mlsanity.com/paper/1803.00401/) (2018)
- [The iNaturalist Species Classification and Detection Dataset](https://mlsanity.com/paper/1707.06642/) (2017)
- [Anchor Loss: Modulating Loss Scale based on Prediction Difficulty](https://mlsanity.com/paper/1909.11155/) (2019)

## Built on by (most cited first, 30 of 84)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [Long-Tailed Recognition via Weight Balancing](https://mlsanity.com/paper/2203.14197/) (2022)
- [Generating images of rare concepts using pre-trained diffusion models](https://mlsanity.com/paper/2304.14530/) (2023)
- [A Survey on Long-Tailed Visual Recognition](https://mlsanity.com/paper/2205.13775/) (2022)
- [VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2111.13579/) (2021)
- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [A Simple Long-Tailed Recognition Baseline via Vision-Language Model](https://mlsanity.com/paper/2111.14745/) (2021)
- [Discovering Objects that Can Move](https://mlsanity.com/paper/2203.10159/) (2022)
- [Norm-guided latent space exploration for text-to-image generation](https://mlsanity.com/paper/2306.08687/) (2023)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [Robust Long-Tailed Learning under Label Noise](https://mlsanity.com/paper/2108.11569/) (2021)
- [How Re-sampling Helps for Long-Tail Learning?](https://mlsanity.com/paper/2310.18236/) (2023)
- [Feedback-guided Data Synthesis for Imbalanced Classification](https://mlsanity.com/paper/2310.00158/) (2023)
- [FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2303.16574/) (2023)
- [ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition](https://mlsanity.com/paper/2308.07815/) (2023)
- [Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data](https://mlsanity.com/paper/2212.13827/) (2022)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Constructing Balance from Imbalance for Long-tailed Image Recognition](https://mlsanity.com/paper/2208.02567/) (2022)
- [A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization](https://mlsanity.com/paper/2310.04752/) (2023)
- [Use Your Head: Improving Long-Tail Video Recognition](https://mlsanity.com/paper/2304.01143/) (2023)
- [Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts](https://mlsanity.com/paper/2308.08810/) (2023)
- [Balanced Product of Calibrated Experts for Long-Tailed Recognition](https://mlsanity.com/paper/2206.05260/) (2022)
- [Re-weighting Based Group Fairness Regularization via Classwise Robust Optimization](https://mlsanity.com/paper/2303.00442/) (2023)
- [Delving into the Trajectory Long-tail Distribution for Muti-object Tracking](https://mlsanity.com/paper/2403.04700/) (2024)
- [Is Self-Supervised Learning More Robust Than Supervised Learning?](https://mlsanity.com/paper/2206.05259/) (2022)
- [Towards Calibrated Hyper-Sphere Representation via Distribution Overlap Coefficient for Long-tailed Learning](https://mlsanity.com/paper/2208.10043/) (2022)
- [Long-tail Recognition via Compositional Knowledge Transfer](https://mlsanity.com/paper/2112.06741/) (2021)
- [LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition](https://mlsanity.com/paper/2305.04536/) (2023)
- [No One Left Behind: Improving the Worst Categories in Long-Tailed Learning](https://mlsanity.com/paper/2303.03630/) (2023)

## Abstract (arXiv)

Real-world data is often unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. To address unbalanced data, most studies try balancing the data, the loss, or the classifier to reduce classification bias towards head classes. Far less attention has been given to the latent representations learned with unbalanced data. We show that the feature extractor part of deep networks suffers greatly from this bias. We propose a new loss based on robustness theory, which encourages the model to learn high-quality representations for both head and tail classes. While the general form of the robustness loss may be hard to compute, we further derive an easy-to-compute upper bound that can be minimized efficiently. This procedure reduces representation bias towards head classes in the feature space and achieves new SOTA results on CIFAR100-LT, ImageNet-LT, and iNaturalist long-tail benchmarks. We find that training with robustness increases recognition accuracy of tail classes while largely maintaining the accuracy of head classes. The new robustness loss can be combined with various classifier balancing techniques and can be applied to representations at several layers of the deep model.