# CIDEr-R: Robust Consensus-based Image Description Evaluation

arXiv: https://arxiv.org/abs/2109.13701 · HTML: https://mlsanity.com/paper/2109.13701/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Captioning Images Taken by People Who Are Blind](https://mlsanity.com/paper/2002.08565/) (2020)
- [Neural Baby Talk](https://mlsanity.com/paper/1803.09845/) (2018)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Areas of Attention for Image Captioning](https://mlsanity.com/paper/1612.01033/) (2016)
- [#PraCegoVer: A Large Dataset for Image Captioning in Portuguese](https://mlsanity.com/paper/2103.11474/) (2021)

## Built on by (most cited first, 22 of 22)

- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation](https://mlsanity.com/paper/2407.02235/) (2024)
- [WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding](https://mlsanity.com/paper/2407.15350/) (2024)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2409.19255/) (2024)
- [4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding](https://mlsanity.com/paper/2503.17827/) (2025)
- [No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning](https://mlsanity.com/paper/2409.03025/) (2024)
- [PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twins](https://mlsanity.com/paper/2508.10906/) (2025)
- [VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions](https://mlsanity.com/paper/2509.25818/) (2025)
- [FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows](https://mlsanity.com/paper/2202.06633/) (2022)
- [ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail](https://mlsanity.com/paper/2503.17044/) (2025)
- [Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation](https://mlsanity.com/paper/2412.15939/) (2024)
- [A request for clarity over the End of Sequence token in the Self-Critical Sequence Training](https://mlsanity.com/paper/2305.12254/) (2023)
- [Inserting Faces inside Captions: Image Captioning with Attention Guided Merging](https://mlsanity.com/paper/2405.02305/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates](https://mlsanity.com/paper/2503.07216/) (2025)
- [Physics-Based Benchmarking Metrics for Multimodal Synthetic Images](https://mlsanity.com/paper/2511.15204/) (2025)
- [Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters](https://mlsanity.com/paper/2602.01390/) (2026)
- [Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2606.29997/) (2026)
- [MUNIChus: Multilingual News Image Captioning Benchmark](https://mlsanity.com/paper/2603.10613/) (2026)
- [InVitroVision: a Multi-Modal AI Model for Automated Description of Embryo Development using Natural Language](https://mlsanity.com/paper/2604.21061/) (2026)

## Abstract (arXiv)

This paper shows that CIDEr-D, a traditional evaluation metric for image description, does not work properly on datasets where the number of words in the sentence is significantly greater than those in the MS COCO Captions dataset. We also show that CIDEr-D has performance hampered by the lack of multiple reference sentences and high variance of sentence length. To bypass this problem, we introduce CIDEr-R, which improves CIDEr-D, making it more flexible in dealing with datasets with high sentence length variance. We demonstrate that CIDEr-R is more accurate and closer to human judgment than CIDEr-D; CIDEr-R is more robust regarding the number of available references. Our results reveal that using Self-Critical Sequence Training to optimize CIDEr-R generates descriptive captions. In contrast, when CIDEr-D is optimized, the generated captions' length tends to be similar to the reference length. However, the models also repeat several times the same word to increase the sentence length.