# VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition

arXiv: https://arxiv.org/abs/2111.13579 · HTML: https://mlsanity.com/paper/2111.13579/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019)
- [Parametric Contrastive Learning](https://mlsanity.com/paper/2107.12028/) (2021)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [Feature Space Augmentation for Long-Tailed Data](https://mlsanity.com/paper/2008.03673/) (2020)

## Built on by (most cited first, 30 of 65)

- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [K-LITE: Learning Transferable Visual Models with External Knowledge](https://mlsanity.com/paper/2204.09222/) (2022)
- [Generating images of rare concepts using pre-trained diffusion models](https://mlsanity.com/paper/2304.14530/) (2023)
- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts](https://mlsanity.com/paper/2309.10019/) (2023)
- [Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions](https://mlsanity.com/paper/2401.02460/) (2024)
- [Norm-guided latent space exploration for text-to-image generation](https://mlsanity.com/paper/2306.08687/) (2023)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2408.04961/) (2024)
- [Exploring Vision-Language Models for Imbalanced Learning](https://mlsanity.com/paper/2304.01457/) (2023)
- [Fill-Up: Balancing Long-Tailed Data with Generative Models](https://mlsanity.com/paper/2306.07200/) (2023)
- [Distilling Large Vision-Language Model with Out-of-Distribution Generalizability](https://mlsanity.com/paper/2307.03135/) (2023)
- [Feedback-guided Data Synthesis for Imbalanced Classification](https://mlsanity.com/paper/2310.00158/) (2023)
- [LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content](https://mlsanity.com/paper/2403.05854/) (2024)
- [Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information](https://mlsanity.com/paper/2211.09807/) (2022)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Improving Image Recognition by Retrieving from Web-Scale Image-Text Data](https://mlsanity.com/paper/2304.05173/) (2023)
- [Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards](https://mlsanity.com/paper/2405.13459/) (2024)
- [Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels](https://mlsanity.com/paper/2305.01160/) (2023)
- [LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition](https://mlsanity.com/paper/2305.04536/) (2023)
- [Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2410.21042/) (2024)
- [DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets](https://mlsanity.com/paper/2404.02900/) (2024)
- [Cap2Aug: Caption guided Image to Image data Augmentation](https://mlsanity.com/paper/2212.05404/) (2022)
- [TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation](https://mlsanity.com/paper/2307.14611/) (2023)
- [Learning from Neighbors: Category Extrapolation for Long-Tail Learning](https://mlsanity.com/paper/2410.15980/) (2024)
- [GeNIe: Generative Hard Negative Images Through Diffusion](https://mlsanity.com/paper/2312.02548/) (2023)
- [Exploring Weight Balancing on Long-Tailed Recognition Problem](https://mlsanity.com/paper/2305.16573/) (2023)
- [SCoRe: Submodular Combinatorial Representation Learning](https://mlsanity.com/paper/2310.00165/) (2023)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)

## Abstract (arXiv)

Deep learning-based models encounter challenges when processing long-tailed data in the real world. Existing solutions usually employ some balancing strategies or transfer learning to deal with the class imbalance problem, based on the image modality. In this work, we present a visual-linguistic long-tailed recognition framework, termed VL-LTR, and conduct empirical studies on the benefits of introducing text modality for long-tailed recognition (LTR). Compared to existing approaches, the proposed VL-LTR has the following merits. (1) Our method can not only learn visual representation from images but also learn corresponding linguistic representation from noisy class-level text descriptions collected from the Internet; (2) Our method can effectively use the learned visual-linguistic representation to improve the visual recognition performance, especially for classes with fewer image samples. We also conduct extensive experiments and set the new state-of-the-art performance on widely-used LTR benchmarks. Notably, our method achieves 77.2% overall accuracy on ImageNet-LT, which significantly outperforms the previous best method by over 17 points, and is close to the prevailing performance training on the full ImageNet. Code is available at https://github.com/ChangyaoTian/VL-LTR.