mlsanityData to 14 Oct 2026

CIDEr-R: Robust Consensus-based Image Description Evaluation

arXiv 2109.13701 ↗First posted 28 Sep 2021cs.CV cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

This paper shows that CIDEr-D, a traditional evaluation metric for image description, does not work properly on datasets where the number of words in the sentence is significantly greater than those in the MS COCO Captions dataset. We also show that CIDEr-D has performance hampered by the lack of multiple reference sentences and high variance of sentence length. To bypass this problem, we introduce CIDEr-R, which improves CIDEr-D, making it more flexible in dealing with datasets with high sentence length variance. We demonstrate that CIDEr-R is more accurate and closer to human judgment than CIDEr-D; CIDEr-R is more robust regarding the number of available references. Our results reveal that using Self-Critical Sequence Training to optimize CIDEr-R generates descriptive captions. In contrast, when CIDEr-D is optimized, the generated captions' length tends to be similar to the reference length. However, the models also repeat several times the same word to increase the sentence length.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

25 of 37 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
13 more
  1. 2016Self-critical Sequence Training for Image Captioningcited by 547
  2. 2018nocaps: novel object captioning at scalecited by 444
  3. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  4. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  5. 2019Attention on Attention for Image Captioningcited by 289
  6. 2020X-Linear Attention Networks for Image Captioningcited by 198
  7. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  8. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
  9. 2020Captioning Images Taken by People Who Are Blindcited by 110
  10. 2018Neural Baby Talkcited by 107
  11. 2020Visual Commonsense R-CNNcited by 106
  12. 2016Areas of Attention for Image Captioningcited by 24
  13. 2021#PraCegoVer: A Large Dataset for Image Captioning in Portuguesecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 10; 2025: 6; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  2. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  3. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  4. 2024The 8th AI City Challengecited by 16
  5. 2024GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpathscited by 15
  6. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  7. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  8. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  9. 2025PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twinscited by 5
  10. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  11. 2022FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flowscited by 4
  12. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
10 more
  1. 2024Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentationcited by 3
  2. 2023A request for clarity over the End of Sequence token in the Self-Critical Sequence Trainingcited by 2
  3. 2024Inserting Faces inside Captions: Image Captioning with Attention Guided Mergingcited by 1
  4. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  5. 2025FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updatescited by 1
  6. 2025Physics-Based Benchmarking Metrics for Multimodal Synthetic Imagescited by 1
  7. 2026Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raterscited by 1
  8. 2026Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluationcited by 1
  9. 2026MUNIChus: Multilingual News Image Captioning Benchmarkcited by 0
  10. 2026InVitroVision: a Multi-Modal AI Model for Automated Description of Embryo Development using Natural Languagecited by 0