# SPICE: Semantic Propositional Image Caption Evaluation

arXiv: https://arxiv.org/abs/1607.08822 · HTML: https://mlsanity.com/paper/1607.08822/

## Key facts

- Cited by 824 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2016.

## Built on by (most cited first, 30 of 824)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://mlsanity.com/paper/2202.04053/) (2022)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment](https://mlsanity.com/paper/2503.09594/) (2025)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)

## Abstract (arXiv)

There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as `which caption-generator best understands colors?' and `can caption-generators count?'