mlsanityData to 7 Oct 2026

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

arXiv 2104.08718 ↗First posted 18 Apr 2021cs.CV cs.CLcited by 2,388

Not on the map. Why?

Cited by 866 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.

Key facts. Cited by 2,388 papers in the map (data through 7 Oct 2026). Builds on 40 of its 75 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

40 of 75 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2018Model Cards for Model Reportingcited by 1,068
28 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2018Object Hallucination in Image Captioningcited by 661
  3. 2019How multilingual is Multilingual BERT?cited by 642
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  6. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  7. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  8. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  9. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  10. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  11. 2016Movie Descriptioncited by 180
  12. 2020With Little Power Comes Great Responsibilitycited by 138
  13. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  14. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 106
  15. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  16. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  17. 2018Discriminability objective for training descriptive captionscited by 67
  18. 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
  19. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  20. 2017Contrastive Learning for Image Captioningcited by 54
  21. 2018Learning to Evaluate Image Captioningcited by 53
  22. 2018Engaging Image Captioning Via Personalitycited by 51
  23. 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
  24. 2018Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Datacited by 37
  25. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  26. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  27. 2020NUBIA: NeUral Based Interchangeability Assessor for Text Generationcited by 34
  28. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 61; 2023: 296; 2024: 603; 2025: 809; 2026: 608.

2,388 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  7. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  10. 2023Adversarial Diffusion Distillationcited by 741
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
38 more
  1. 2021Extract Free Dense Labels from CLIPcited by 580
  2. 2023Zero-shot Image-to-Image Translationcited by 503
  3. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  6. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  8. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  10. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  11. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  12. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  13. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  14. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  15. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  16. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  17. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  18. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  19. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  20. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  21. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  22. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  23. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  24. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  25. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  26. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  27. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  28. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  29. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  30. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  31. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  32. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  33. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  34. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  35. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  36. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  37. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  38. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182

50 most cited of 2,388 All 2,388 (JSON)