mlsanityData to 7 Oct 2026

CLAIR: Evaluating Image Captions with Large Language Models

arXiv 2310.12971 ↗First posted 19 Oct 2023cs.CV cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available at https://davidmchan.github.io/clair/

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

23 of 42 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
11 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2018Object Hallucination in Image Captioningcited by 661
  3. 2016Self-critical Sequence Training for Image Captioningcited by 544
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  5. 2016Visual Storytellingcited by 176
  6. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  7. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  8. 2018Learning to Evaluate Image Captioningcited by 53
  9. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  10. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  11. 2020NUBIA: NeUral Based Interchangeability Assessor for Text Generationcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 18; 2026: 12.

43 papers.

Sort:

By area:

  1. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  2. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  3. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  4. 2025Lost in Translation, Found in Context: Sign Language Translation with Contextual Cuescited by 25
  5. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  6. 2025Benchmarking Visual Language Models on Standardized Visualization Literacy Testscited by 20
  7. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  8. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understandingcited by 16
  9. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
  10. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  11. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 13
  12. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictionscited by 11
31 more
  1. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  2. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  3. 2024Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Modelscited by 10
  4. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  5. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  6. 2024Wolf: Dense Video Captioning with a World Summarization Frameworkcited by 7
  7. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  8. 2025Multi-LLM Collaborative Caption Generation in Scientific Documentscited by 6
  9. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5
  10. 2024CLAIR-A: Leveraging Large Language Models to Judge Audio Captionscited by 5
  11. 2025From Panels to Prose: Generating Literary Narratives from Comicscited by 5
  12. 2025CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioningcited by 5
  13. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  14. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  15. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  16. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  17. 2026BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observationcited by 2
  18. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  19. 2026MLLM-as-a-Judge Exhibits Model Preference Biascited by 1
  20. 2026Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluationcited by 1
  21. 2024Training-Free Action Recognition and Goal Inference with Dynamic Frame Selectioncited by 0
  22. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  23. 2025Disentangling Fine-Tuning from Pre-Training in Visual Captioning with Hybrid Markov Logiccited by 0
  24. 2025IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truthcited by 0
  25. 2026Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Datasetcited by 0
  26. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  27. 2026DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarioscited by 0
  28. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0
  29. 2026PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Predictioncited by 0
  30. 2026Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Datasetcited by 0
  31. 2026Spot-the-shift: Evaluating Grounded Image Difference Captioning of Long-term Changescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  2. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  3. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  4. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  5. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
  6. 2024Benchmarking and Improving Detail Image Caption
  7. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  8. 2023CapsFusion: Rethinking Image-Text Data at Scale