mlsanityData to 7 Oct 2026

SPICE: Semantic Propositional Image Caption Evaluation

arXiv 1607.08822 ↗First posted 29 Jul 2016cs.CV cs.CLcited by 820

Not on the map. Why?

Cited by 136 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as `which caption-generator best understands colors?' and `can caption-generators count?'

Key facts. Cited by 820 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 0; 2021: 141; 2022: 141; 2023: 145; 2024: 176; 2025: 137; 2026: 79.

820 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
38 more
  1. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  4. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  5. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  7. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  8. 2023Video Understanding with Large Language Models: A Surveycited by 203
  9. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  10. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  11. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  12. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  13. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  14. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  15. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  16. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  17. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  18. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  19. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  20. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  21. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  22. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  23. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  24. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  25. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  26. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  27. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  28. 2021Causal Attention for Vision-Language Taskscited by 111
  29. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  30. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  31. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 106
  32. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  33. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  34. 2022Language Models are General-Purpose Interfacescited by 96
  35. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  36. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  37. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  38. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89

50 most cited of 820 All 820 (JSON)