# QACE: Asking Questions to Evaluate an Image Caption

arXiv: https://arxiv.org/abs/2108.12560 · HTML: https://mlsanity.com/paper/2108.12560/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 28 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge](https://mlsanity.com/paper/1511.03292/) (2015)
- [CapWAP: Captioning with a Purpose](https://mlsanity.com/paper/2011.04264/) (2020)

## Built on by (most cited first, 15 of 15)

- [Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models](https://mlsanity.com/paper/2409.10695/) (2024)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics](https://mlsanity.com/paper/2205.10646/) (2022)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning](https://mlsanity.com/paper/2508.06125/) (2025)
- [Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2306.16478/) (2023)
- [MaXM: Towards Multilingual Visual Question Answering](https://mlsanity.com/paper/2209.05401/) (2022)
- [Reference-free Evaluation Metrics for Text Generation: A Survey](https://mlsanity.com/paper/2501.12011/) (2025)
- [VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation](https://mlsanity.com/paper/2505.23484/) (2025)
- [MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification](https://mlsanity.com/paper/2205.12394/) (2022)
- [ContextRef: Evaluating Referenceless Metrics For Image Description Generation](https://mlsanity.com/paper/2309.11710/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [OAD-Promoter: Enhancing Zero-shot VQA using Large Language Models with Object Attribute Description](https://mlsanity.com/paper/2511.12131/) (2025)
- [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://mlsanity.com/paper/2609.09973/) (2026)

## Abstract (arXiv)

In this paper, we propose QACE, a new metric based on Question Answering for Caption Evaluation. QACE generates questions on the evaluated caption and checks its content by asking the questions on either the reference caption or the source image. We first develop QACE-Ref that compares the answers of the evaluated caption to its reference, and report competitive results with the state-of-the-art metrics. To go further, we propose QACE-Img, which asks the questions directly on the image, instead of reference. A Visual-QA system is necessary for QACE-Img. Unfortunately, the standard VQA models are framed as a classification among only a few thousand categories. Instead, we propose Visual-T5, an abstractive VQA system. The resulting metric, QACE-Img is multi-modal, reference-less, and explainable. Our experiments show that QACE-Img compares favorably w.r.t. other reference-less metrics. We will release the pre-trained models to compute QACE.