mlsanityData to 7 Oct 2026

Teaching CLIP to Count to Ten

arXiv 2302.12066 ↗First posted 23 Feb 2023cs.CVcited by 194

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (VLMs), such as CLIP, learn rich joint image-text representations, facilitating advances in numerous downstream tasks, including zero-shot classification and text-to-image generation. Nevertheless, existing VLMs exhibit a prominent well-documented limitation - they fail to encapsulate compositional concepts such as counting. We introduce a simple yet effective method to improve the quantitative understanding of VLMs, while maintaining their overall performance on common benchmarks. Specifically, we propose a new counting-contrastive loss used to finetune a pre-trained VLM in tandem with its original objective. Our counting loss is deployed over automatically-created counterfactual examples, each consisting of an image and a caption containing an incorrect object count. For example, an image depicting three dogs is paired with the caption "Six dogs playing in the yard". Our loss encourages discrimination between the correct caption and its counterfactual variant which serves as a hard negative example. To the best of our knowledge, this work is the first to extend CLIP's capabilities to object counting. Furthermore, we introduce "CountBench" - a new image-text counting benchmark for evaluating a model's understanding of object counting. We demonstrate a significant improvement over state-of-the-art baseline models on this task. Finally, we leverage our count-aware CLIP model for image retrieval and text-conditioned image generation, demonstrating that our model can produce specific counts of objects more reliably than existing ones.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 74 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

54 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
38 more
  1. 2019Searching for MobileNetV3cited by 1,984
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  10. 2017Deep Image Priorcited by 1,097
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  12. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  13. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2022Language-driven Semantic Segmentationcited by 711
  16. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  17. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  18. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  20. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  22. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  23. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  24. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  25. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  26. 2018TallyQA: Answering Complex Counting Questionscited by 152
  27. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  28. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  29. 2021Learning to Compose Visual Relationscited by 72
  30. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  31. 2022Disentangling visual and written concepts in CLIPcited by 57
  32. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 50
  33. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  34. 2018Learning to Count Objects in Natural Images for Visual Question Answeringcited by 40
  35. 2021Image-Based CLIP-Guided Essence Transfercited by 39
  36. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  37. 2022No Token Left Behind: Explainability-Aided Image Classification and Generationcited by 25
  38. 2021Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examplescited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 37; 2025: 58; 2026: 75.

194 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2025Qwen3-Omni Technical Reportcited by 451
  4. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  7. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  8. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  9. 2026Qwen3.5-Omni Technical Reportcited by 146
  10. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 128
  11. 2025MiMo-VL Technical Reportcited by 106
  12. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generationcited by 93
38 more
  1. 2023CLIP-Count: Towards Text-Guided Zero-Shot Object Countingcited by 82
  2. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  3. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  4. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  5. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  6. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
  7. 2024CountGD: Multi-Modal Open-World Countingcited by 59
  8. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  9. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  10. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
  11. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  12. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  13. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  14. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  15. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  16. 2023Open-world Text-specified Object Countingcited by 41
  17. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  18. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  19. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  20. 2023Alchemist: Parametric Control of Material Properties with Diffusion Modelscited by 38
  21. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  22. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  23. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  24. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  25. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  26. 2026STEP3-VL-10B Technical Reportcited by 28
  27. 2023Counting Guidance for High Fidelity Text-to-Image Synthesiscited by 24
  28. 2023SPIRE: Semantic Prompt-Driven Image Restorationcited by 24
  29. 2025LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extractioncited by 24
  30. 2025LongCat-Flash-Omni Technical Reportcited by 23
  31. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  32. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  33. 2024Teach CLIP to Develop a Number Sense for Ordinal Regressioncited by 20
  34. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  35. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  36. 2024Nomic Embed Vision: Expanding the Latent Spacecited by 19
  37. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
  38. 2024When Does Perceptual Alignment Benefit Vision Representations?cited by 18

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models
  2. 2025Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models
  3. 2025CountQA: How Well Do MLLMs Count in the Wild?
  4. 2026Counting to Four is still a Chore for VLMs
  5. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
  6. 2025Object Counting with GPT-4o and GPT-5: A Comparative Study
  7. 2025T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting
  8. 2026UNICBench: UNIfied Counting Benchmark for MLLM