# Exploring Nearest Neighbor Approaches for Image Captioning

arXiv: https://arxiv.org/abs/1505.04467 · HTML: https://mlsanity.com/paper/1505.04467/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2015.

## Built on by (most cited first, 19 of 19)

- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)
- [NICE: CVPR 2023 Challenge on Zero-shot Image Captioning](https://mlsanity.com/paper/2309.01961/) (2023)
- [Rethinking Nearest Neighbors for Visual Classification](https://mlsanity.com/paper/2112.08459/) (2021)
- [Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis](https://mlsanity.com/paper/2303.00703/) (2023)
- [FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows](https://mlsanity.com/paper/2202.06633/) (2022)
- [Set Prediction Guided by Semantic Concepts for Diverse Video Captioning](https://mlsanity.com/paper/2312.15720/) (2023)
- [3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset](https://mlsanity.com/paper/2404.18413/) (2024)
- [Diverse Image Captioning with Grounded Style](https://mlsanity.com/paper/2205.01813/) (2022)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification](https://mlsanity.com/paper/2303.15016/) (2023)
- [Explaining the Success of Nearest Neighbor Methods in Prediction](https://mlsanity.com/paper/2502.15900/) (2025)
- [Pixels to Prose: Understanding the art of Image Captioning](https://mlsanity.com/paper/2408.15714/) (2024)
- [A Hybrid Model for Combining Neural Image Caption and k-Nearest Neighbor Approach for Image Captioning](https://mlsanity.com/paper/2105.03826/) (2021)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)
- [Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusion](https://mlsanity.com/paper/2603.22946/) (2026)
- [NeuroSymbEAD: A Large Scale Neuro-Symbolic Caption Dataset for Omni-Directional Embodied Autonomous Driving](https://mlsanity.com/paper/2609.16919/) (2026)

## Abstract (arXiv)

We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We select a caption for the query image by finding the caption that best represents the "consensus" of the set of candidate captions gathered from the nearest neighbor images. When measured by automatic evaluation metrics on the MS COCO caption evaluation server, these approaches perform as well as many recent approaches that generate novel captions. However, human studies show that a method that generates novel captions is still preferred over the nearest neighbor approach.