# Group Relative Policy Optimization for Image Captioning

arXiv: https://arxiv.org/abs/2503.01333 · HTML: https://mlsanity.com/paper/2503.01333/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)

## Built on by (most cited first, 9 of 9)

- [Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach](https://mlsanity.com/paper/2503.21819/) (2025)
- [GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.17939/) (2025)
- [CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation](https://mlsanity.com/paper/2507.06013/) (2025)
- [RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment](https://mlsanity.com/paper/2507.22580/) (2025)
- [TaxonRL: Reinforcement Learning with Intermediate Rewards for Interpretable Fine-Grained Visual Reasoning](https://mlsanity.com/paper/2603.04380/) (2026)
- [FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures](https://mlsanity.com/paper/2601.08026/) (2026)
- [BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards](https://mlsanity.com/paper/2606.28707/) (2026)
- [Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment](https://mlsanity.com/paper/2608.19825/) (2026)
- [Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraint](https://mlsanity.com/paper/2609.19812/) (2026)

## Abstract (arXiv)

Image captioning tasks usually use two-stage training to complete model optimization. The first stage uses cross-entropy as the loss function for optimization, and the second stage uses self-critical sequence training (SCST) for reinforcement learning optimization. However, the SCST algorithm has certain defects. SCST relies only on a single greedy decoding result as a baseline. If the model itself is not stable enough, the greedy decoding result may be relatively worst, which will lead to a high variance of advantage estimation, further leading to unstable policy updates. In addition, SCST only compares one sampling result with the greedy decoding result, and the generation diversity is limited, which may fall into a local optimum. In this paper, we propose using the latest Group Relative Policy Optimization (GRPO) reinforcement learning algorithm as an optimization solution for the second stage. GRPO generates multiple candidate captions for the input image and then continuously optimizes the model through intragroup comparison. By constraining the amplitude of policy updates and KL divergence, the stability of the model during training is greatly guaranteed. In addition, compared to SCST, which only samples one answer, GRPO samples and generates multiple answers. Multiple candidate answers in the group cover a wider solution space. Combined with KL divergence constraints, GRPO can improve diversity while ensuring model stability. The code for this article is available at https://github.com/liangxu-one/ms-models/tree/image_caption_grpo/research/arxiv_papers/Image_Caption_GRPO.