# Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data

arXiv: https://arxiv.org/abs/2507.08610 · HTML: https://mlsanity.com/paper/2507.08610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019)
- [Towards Unsupervised Image Captioning with Shared Multimodal Embeddings](https://mlsanity.com/paper/1908.09317/) (2019)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Emergent Communication of Generalizations](https://mlsanity.com/paper/2106.02668/) (2021)
- [Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data](https://mlsanity.com/paper/1803.08314/) (2018)
- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner](https://mlsanity.com/paper/1705.00930/) (2017)
- [Recurrent Relational Memory Network for Unsupervised Image Captioning](https://mlsanity.com/paper/2006.13611/) (2020)
- [Explain Images with Multimodal Recurrent Neural Networks](https://mlsanity.com/paper/1410.1090/) (2014)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [Learning a Recurrent Visual Representation for Image Caption Generation](https://mlsanity.com/paper/1411.5654/) (2014)
- [Partially-Supervised Image Captioning](https://mlsanity.com/paper/1806.06004/) (2018)
- [Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segments](https://mlsanity.com/paper/2311.04453/) (2023)
- [Self-Supervised Image Captioning with CLIP](https://mlsanity.com/paper/2306.15111/) (2023)

## Built on by (most cited first, 1 of 1)

- [Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions](https://mlsanity.com/paper/2606.08081/) (2026)

## Abstract (arXiv)

Image captioning is an important problem in developing various AI systems, and these tasks require large volumes of annotated images to train the models. Since all existing labelled datasets are already used for training the large Vision Language Models (VLMs), it becomes challenging to improve the performance of the same. Considering this, it is essential to consider the unsupervised image captioning performance, which remains relatively under-explored. To that end, we propose LoGIC (Lewis Communication Game for Image Captioning), a Multi-agent Reinforcement Learning game. The proposed method consists of two agents, a 'speaker' and a 'listener', with the objective of learning a strategy for communicating in natural language. We train agents in the cooperative common-reward setting using the GRPO algorithm and show that improvement in image captioning performance emerges as a consequence of the agents learning to play the game. We show that using pre-trained VLMs as the 'speaker' and Large Language Model (LLM) for language understanding in the 'listener', we achieved a 46 BLEU score after fine-tuning using LoGIC without additional labels, a 2 units advantage in absolute metrics compared to the 44 BLEU score of the vanilla VLM. Additionally, we replace the VLM from the 'speaker' with lightweight components: (i) a ViT for image perception and (ii) a GPT2 language generation, and train them from scratch using LoGIC, obtaining a 31 BLEU score in the unsupervised setting, a 10 points advantage over existing unsupervised image-captioning methods.