mlsanityData to 7 Oct 2026

The Neglected Tails in Vision-Language Models

arXiv 2401.12425 ↗First posted 23 Jan 2024cs.CV cs.CL cs.LGcited by 63

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) excel in zero-shot recognition but their performance varies greatly across different visual concepts. For example, although CLIP achieves impressive accuracy on ImageNet (60-80%), its performance drops below 10% for more than ten concepts like night snake, presumably due to their limited presence in the pretraining data. However, measuring the frequency of concepts in VLMs' large-scale datasets is challenging. We address this by using large language models (LLMs) to count the number of pretraining texts that contain synonyms of these concepts. Our analysis confirms that popular datasets, such as LAION, exhibit a long-tailed concept distribution, yielding biased performance in VLMs. We also find that downstream applications of VLMs, including visual chatbots (e.g., GPT-4V) and text-to-image models (e.g., Stable Diffusion), often fail to recognize or generate images of rare concepts identified by our method. To mitigate the imbalanced performance of zero-shot VLMs, we propose REtrieval-Augmented Learning (REAL). First, instead of prompting VLMs using the original class names, REAL uses their most frequent synonyms found in pretraining texts. This simple change already outperforms costly human-engineered and LLM-enriched prompts over nine benchmark datasets. Second, REAL trains a linear classifier on a small yet balanced set of pretraining data retrieved using concept synonyms. REAL surpasses the previous zero-shot SOTA, using 400x less storage and 10,000x less training time!

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 50 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

41 of 50 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
29 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  9. 2018Do Better ImageNet Models Transfer Better?cited by 560
  10. 2022Visual Classification via Description from Large Language Modelscited by 331
  11. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  12. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  13. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 165
  14. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  15. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  16. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  17. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  18. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  19. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  20. 2020Improving Image Captioning with Better Use of Captionscited by 84
  21. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  22. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  23. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  24. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  25. 2023Internet Explorer: Targeted Representation Learning on the Open Webcited by 21
  26. 2023Prompting Scientific Names for Zero-Shot Species Recognitioncited by 14
  27. 2023Neural Priming for Sample-Efficient Adaptationcited by 12
  28. 2023Investigating the Limitation of CLIP Models: The Worst-Performing Categoriescited by 12
  29. 2022Task Bias in Vision-Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 32; 2026: 12.

63 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2024An Introduction to Vision-Language Modelingcited by 175
  3. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  4. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  5. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  6. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  7. 2025Towards Understanding Camera Motions in Any Videocited by 62
  8. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  9. 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaborationcited by 33
  10. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  11. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  12. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
38 more
  1. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  2. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  3. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  4. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Modelscited by 20
  5. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 18
  6. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  7. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  8. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  9. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  10. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  11. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Modelscited by 10
  12. 2025From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibrationcited by 8
  13. 2025CuRe: Cultural Gaps in the Long Tail of Text-to-Image Systemscited by 8
  14. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  15. 2025Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognitioncited by 7
  16. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  17. 2025Complex LLM Planning via Automated Heuristics Discoverycited by 6
  18. 2025Solving Instance Detection from an Open-World Perspectivecited by 6
  19. 2025DocVXQA: Context-Aware Visual Explanations for Document Question Answeringcited by 5
  20. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  21. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 5
  22. 2025Expertized Caption Auto-Enhancement for Video-Text Retrievalcited by 4
  23. 2025Attention to the Burstiness in Visual Prompt Tuning!cited by 4
  24. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  25. 2024UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmarkcited by 3
  26. 2024A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safetycited by 3
  27. 20256 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Modelscited by 3
  28. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  29. 2026Learnability-Informed Fine-Tuning of Diffusion Language Modelscited by 3
  30. 2024Better Language Models Exhibit Higher Visual Alignmentcited by 2
  31. 2025Active Learning via Vision-Language Model Adaptation with Open Datacited by 2
  32. 2025Enabling Validation for Robust Few-Shot Recognitioncited by 2
  33. 2025Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similaritycited by 2
  34. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  35. 2026Adaptive Debiasing Tsallis Entropy for Test-Time Adaptationcited by 2
  36. 2025MIAT: Maneuver-Intention-Aware Transformer for Spatio-Temporal Trajectory Predictioncited by 1
  37. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  38. 2026Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrievalcited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Revisiting the Role of Language Priors in Vision-Language Models
  2. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  3. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  4. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  5. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Models
  6. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection