# Learning Imbalanced Data with Vision Transformers

arXiv: https://arxiv.org/abs/2212.02015 · HTML: https://mlsanity.com/paper/2212.02015/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 82 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 30 of 70)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)

## Built on by (most cited first, 30 of 34)

- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [Neural Degradation Representation Learning for All-In-One Image Restoration](https://mlsanity.com/paper/2310.12848/) (2023)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards](https://mlsanity.com/paper/2405.13459/) (2024)
- [Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2410.21042/) (2024)
- [DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets](https://mlsanity.com/paper/2404.02900/) (2024)
- [JointViT: Modeling Oxygen Saturation Levels with Joint Supervision on Long-Tailed OCTA](https://mlsanity.com/paper/2404.11525/) (2024)
- [NeurIPT: Foundation Model for Neural Interfaces](https://mlsanity.com/paper/2510.16548/) (2025)
- [Learning from Neighbors: Category Extrapolation for Long-Tail Learning](https://mlsanity.com/paper/2410.15980/) (2024)
- [GeNIe: Generative Hard Negative Images Through Diffusion](https://mlsanity.com/paper/2312.02548/) (2023)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [Resilient Contrastive Pre-training under Non-Stationary Drift](https://mlsanity.com/paper/2502.07620/) (2025)
- [Fractal Calibration for long-tailed object detection](https://mlsanity.com/paper/2410.11774/) (2024)
- [LIFT+: Lightweight Fine-Tuning for Long-Tail Learning](https://mlsanity.com/paper/2504.13282/) (2025)
- [Generative Data Mining with Longtail-Guided Diffusion](https://mlsanity.com/paper/2502.01980/) (2025)
- [Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification](https://mlsanity.com/paper/2505.08173/) (2025)
- [LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions](https://mlsanity.com/paper/2508.15688/) (2025)
- [IMWA: Iterative Model Weight Averaging Benefits Class-Imbalanced Learning Tasks](https://mlsanity.com/paper/2404.16331/) (2024)
- [Text-Guided Mixup Towards Long-Tailed Image Categorization](https://mlsanity.com/paper/2409.03583/) (2024)
- [Rethinking the Bias of Foundation Model under Long-tailed Distribution](https://mlsanity.com/paper/2501.15955/) (2025)
- [BCE vs. CE in Deep Feature Learning](https://mlsanity.com/paper/2505.05813/) (2025)
- [LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups](https://mlsanity.com/paper/2511.10683/) (2025)
- [RICASSO: Reinforced Imbalance Learning with Class-Aware Self-Supervised Outliers Exposure](https://mlsanity.com/paper/2410.10548/) (2024)
- [PI-H2T: Enhancing Long-Tailed Visual Recognition with Permutation-Invariant and Head-to-Tail Feature Fusion](https://mlsanity.com/paper/2506.00625/) (2025)
- [Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognition](https://mlsanity.com/paper/2507.12807/) (2025)
- [Efficient Long-Tail Learning in Latent Space by sampling Synthetic Data](https://mlsanity.com/paper/2509.15859/) (2025)
- [Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classification](https://mlsanity.com/paper/2602.01633/) (2026)
- [Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains](https://mlsanity.com/paper/2602.02841/) (2026)
- [Image-based Joint-level Detection for Inflammation in Rheumatoid Arthritis from Small and Imbalanced Data](https://mlsanity.com/paper/2602.14365/) (2026)

## Abstract (arXiv)

The real-world data tends to be heavily imbalanced and severely skew the data-driven deep neural networks, which makes Long-Tailed Recognition (LTR) a massive challenging task. Existing LTR methods seldom train Vision Transformers (ViTs) with Long-Tailed (LT) data, while the off-the-shelf pretrain weight of ViTs always leads to unfair comparisons. In this paper, we systematically investigate the ViTs' performance in LTR and propose LiVT to train ViTs from scratch only with LT data. With the observation that ViTs suffer more severe LTR problems, we conduct Masked Generative Pretraining (MGP) to learn generalized features. With ample and solid evidence, we show that MGP is more robust than supervised manners. In addition, Binary Cross Entropy (BCE) loss, which shows conspicuous performance with ViTs, encounters predicaments in LTR. We further propose the balanced BCE to ameliorate it with strong theoretical groundings. Specially, we derive the unbiased extension of Sigmoid and compensate extra logit margins to deploy it. Our Bal-BCE contributes to the quick convergence of ViTs in just a few epochs. Extensive experiments demonstrate that with MGP and Bal-BCE, LiVT successfully trains ViTs well without any additional data and outperforms comparable state-of-the-art methods significantly, e.g., our ViT-B achieves 81.0% Top-1 accuracy in iNaturalist 2018 without bells and whistles. Code is available at https://github.com/XuZhengzhuo/LiVT.