mlsanityData to 14 Oct 2026

JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models

arXiv 2311.04192 ↗First posted 7 Nov 2023cs.CV cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR. However, such n-gram-based metrics have been shown to correlate poorly with human evaluation, leading to the proposal of alternative metrics such as SPICE for English; however, no equivalent metrics have been established for other languages. Therefore, in this study, we propose an automatic evaluation metric called JaSPICE, which evaluates Japanese captions based on scene graphs. The proposed method generates a scene graph from dependencies and the predicate-argument structure, and extends the graph using synonyms. We conducted experiments employing 10 image captioning models trained on STAIR Captions and PFN-PIC and constructed the Shichimi dataset, which contains 103,170 human evaluations. The results showed that our metric outperformed the baseline metrics for the correlation coefficient with the human evaluation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

30 of 51 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  11. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  12. 2018Image Generation from Scene Graphscited by 324
18 more
  1. 2019Image Captioning: Transforming Objects into Wordscited by 171
  2. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  3. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  4. 2020Captioning Images Taken by People Who Are Blindcited by 110
  5. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  6. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  7. 2019Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrievalcited by 66
  8. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 62
  9. 2019BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detectioncited by 62
  10. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  11. 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
  12. 2022Reasoning with Scene Graphs for Robot Planning under Partial Observabilitycited by 25
  13. 2019JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpuscited by 21
  14. 2020Understanding Guided Image Captioning Performance across Domainscited by 21
  15. 2021Open-domain clarification question generation without question examplescited by 20
  16. 2020Alleviating the Burden of Labeling: Sentence Generation by Attention Branch Encoder-Decoder Networkcited by 6
  17. 2019Multimodal Attention Branch Network for Perspective-Free Sentence Generationcited by 4
  18. 2021Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  2. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  3. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  4. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  5. 2025LLM-Free Image Captioning Evaluation in Reference-Flexible Settingscited by 2
  6. 2026Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluationcited by 1