# Quality Estimation for Image Captions Based on Large-scale Human Evaluations

arXiv: https://arxiv.org/abs/1909.03396 · HTML: https://mlsanity.com/paper/1909.03396/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2019.

## Built on by (most cited first, 17 of 17)

- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning](https://mlsanity.com/paper/2106.14019/) (2021)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [#PraCegoVer: A Large Dataset for Image Captioning in Portuguese](https://mlsanity.com/paper/2103.11474/) (2021)
- [Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions](https://mlsanity.com/paper/2507.15692/) (2025)
- [On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation](https://mlsanity.com/paper/2602.06423/) (2026)
- [NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels](https://mlsanity.com/paper/2110.06827/) (2021)
- [Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents](https://mlsanity.com/paper/2502.19917/) (2025)
- [L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training](https://mlsanity.com/paper/2507.08710/) (2025)
- [REST: REtrieve & Self-Train for generative action recognition](https://mlsanity.com/paper/2209.15000/) (2022)
- [Bias Begets Bias: The Impact of Biased Embeddings on Diffusion Models](https://mlsanity.com/paper/2409.09569/) (2024)
- [ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models](https://mlsanity.com/paper/2407.12877/) (2024)
- [A Deep Learning Approach for Augmenting Perceptional Understanding of Histopathology Images](https://mlsanity.com/paper/2503.06894/) (2025)
- [MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation](https://mlsanity.com/paper/2602.09624/) (2026)
- [DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching](https://mlsanity.com/paper/2606.31980/) (2026)

## Abstract (arXiv)

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.