# The Solution for the CVPR2024 NICE Image Captioning Challenge

arXiv: https://arxiv.org/abs/2404.12739 · HTML: https://mlsanity.com/paper/2404.12739/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Dual-Level Collaborative Transformer for Image Captioning](https://mlsanity.com/paper/2101.06462/) (2021)
- [Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning](https://mlsanity.com/paper/2110.11767/) (2021)
- [Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning](https://mlsanity.com/paper/2212.13563/) (2022)
- [Improving Image Recognition by Retrieving from Web-Scale Image-Text Data](https://mlsanity.com/paper/2304.05173/) (2023)
- [Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioning](https://mlsanity.com/paper/2312.08865/) (2023)
- [Self-Enhancement Improves Text-Image Retrieval in Foundation Visual-Language Models](https://mlsanity.com/paper/2306.06691/) (2023)
- [The Solution for the CVPR2023 NICE Image Captioning Challenge](https://mlsanity.com/paper/2310.06879/) (2023)

## Built on by (most cited first, 1 of 1)

- [Rethinking the Role of LLMs in Time Series Forecasting](https://mlsanity.com/paper/2602.14744/) (2026)

## Abstract (arXiv)

This report introduces a solution to the Topic 1 Zero-shot Image Captioning of 2024 NICE : New frontiers for zero-shot Image Captioning Evaluation. In contrast to NICE 2023 datasets, this challenge involves new annotations by humans with significant differences in caption style and content. Therefore, we enhance image captions effectively through retrieval augmentation and caption grading methods. At the data level, we utilize high-quality captions generated by image caption models as training data to address the gap in text styles. At the model level, we employ OFA (a large-scale visual-language pre-training model based on handcrafted templates) to perform the image captioning task. Subsequently, we propose caption-level strategy for the high-quality caption data generated by the image caption models and integrate them with retrieval augmentation strategy into the template to compel the model to generate higher quality, more matching, and semantically enriched captions based on the retrieval augmentation prompts. Our approach achieves a CIDEr score of 234.11.