# IC3: Image Captioning by Committee Consensus

arXiv: https://arxiv.org/abs/2302.01328 · HTML: https://mlsanity.com/paper/2302.01328/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 99 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 30 of 57)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://mlsanity.com/paper/1611.06607/) (2016)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)

## Built on by (most cited first, 20 of 20)

- [ALOHa: A New Measure for Hallucination in Captioning Models](https://mlsanity.com/paper/2404.02904/) (2024)
- [Segment and Caption Anything](https://mlsanity.com/paper/2312.00869/) (2023)
- [Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation](https://mlsanity.com/paper/2404.19752/) (2024)
- [VLRM: Vision-Language Models act as Reward Models for Image Captioning](https://mlsanity.com/paper/2404.01911/) (2024)
- [Guiding Image Captioning Models Toward More Specific Captions](https://mlsanity.com/paper/2307.16686/) (2023)
- [Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition](https://mlsanity.com/paper/2409.09785/) (2024)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment](https://mlsanity.com/paper/2406.13912/) (2024)
- [Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation](https://mlsanity.com/paper/2507.16716/) (2025)
- [Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation](https://mlsanity.com/paper/2408.01363/) (2024)
- [Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization](https://mlsanity.com/paper/2408.14547/) (2024)
- ["It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models](https://mlsanity.com/paper/2511.08917/) (2025)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences](https://mlsanity.com/paper/2507.19362/) (2025)
- [A Comprehensive Analysis of Real-World Image Captioning and Scene Identification](https://mlsanity.com/paper/2308.02833/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions](https://mlsanity.com/paper/2504.08531/) (2025)
- [Fluent and Accurate Image Captioning with a Self-Trained Reward Model](https://mlsanity.com/paper/2408.16827/) (2024)
- [Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment](https://mlsanity.com/paper/2608.19825/) (2026)
- [When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models](https://mlsanity.com/paper/2609.35272/) (2026)

## Abstract (arXiv)

If you ask a human to describe an image, they might do so in a thousand different ways. Traditionally, image captioning models are trained to generate a single "best" (most like a reference) image caption. Unfortunately, doing so encourages captions that are "informationally impoverished," and focus on only a subset of the possible details, while ignoring other potentially useful information in the scene. In this work, we introduce a simple, yet novel, method: "Image Captioning by Committee Consensus" (IC3), designed to generate a single caption that captures high-level details from several annotator viewpoints. Humans rate captions produced by IC3 at least as helpful as baseline SOTA models more than two thirds of the time, and IC3 can improve the performance of SOTA automated recall systems by up to 84%, outperforming single human-generated reference captions, and indicating significant improvements over SOTA approaches for visual description. Code is available at https://davidmchan.github.io/caption-by-committee/