mlsanityData to 14 Oct 2026

PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning

arXiv 2303.08389 ↗First posted 15 Mar 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3, 000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, proving that PR-MCS is highly robust to lexical perturbations.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

36 of 54 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
24 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  3. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  5. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  9. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  10. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  11. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  12. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  13. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  14. 2019Learning from Synthetic Data for Crowd Counting in the Wildcited by 148
  15. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  16. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  17. 2020Captioning Images Taken by People Who Are Blindcited by 110
  18. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  19. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  20. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  21. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  22. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  23. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  24. 2019VIFIDEL: Evaluating the Visual Fidelity of Image Descriptionscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  2. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  3. 2023ContextRef: Evaluating Referenceless Metrics For Image Description Generationcited by 3
  4. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 2
  5. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1