# GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval

arXiv: https://arxiv.org/abs/2509.23370 · HTML: https://mlsanity.com/paper/2509.23370/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [Meta CLIP 2: A Worldwide Scaling Recipe](https://mlsanity.com/paper/2507.22062/) (2025)
- [Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features](https://mlsanity.com/paper/2308.11485/) (2023)
- [Fluency-Guided Cross-Lingual Image Captioning](https://mlsanity.com/paper/1708.04390/) (2017)
- [Transductive Zero-Shot and Few-Shot CLIP](https://mlsanity.com/paper/2405.18437/) (2024)
- [LoTLIP: Improving Language-Image Pre-training for Long Text Understanding](https://mlsanity.com/paper/2410.05249/) (2024)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach](https://mlsanity.com/paper/2406.03411/) (2024)
- [Parameter-efficient Tuning of Large-scale Multimodal Foundation Model](https://mlsanity.com/paper/2305.08381/) (2023)
- [FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs](https://mlsanity.com/paper/2504.01916/) (2025)
- [Scaling Pre-training to One Hundred Billion Data for Vision Language Models](https://mlsanity.com/paper/2502.07617/) (2025)
- [Learning Clustering-based Prototypes for Compositional Zero-shot Learning](https://mlsanity.com/paper/2502.06501/) (2025)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [Hierarchical Cross-modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.14976/) (2025)
- [CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation](https://mlsanity.com/paper/2407.01081/) (2024)
- [Finetuning CLIP to Reason about Pairwise Differences](https://mlsanity.com/paper/2409.09721/) (2024)

## Built on by (most cited first, 1 of 1)

- [Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval](https://mlsanity.com/paper/2610.11666/) (2026)

## Abstract (arXiv)

The CLIP model has established itself as a cornerstone of large-scale retrieval systems. However, its performance often degrades under distributional shifts such as multilingual, long-form, or multimodal queries. To avoid the prohibitive costs associated with retriever retraining or corpus re-embedding, we propose GRAPE (Grouped Ranking-Aware Policy Optimization Enhancement), a plug-and-play approach that leverages LLM-based query rewriting to bridge these gaps. Unlike existing methods that lack explicit supervision, GRAPE integrates ranking signals into the rewriting LLM via Grouped Relative Policy Optimization (GRPO), ensuring rewritten queries are better aligned with the frozen retriever's latent distribution. Crucially, we identify a score inflation phenomenon in naive similarity-based finetuning - where irrelevant candidates receive indiscriminately high scores - and mitigate it with a novel corpus-relative ranking-based reward. Extensive experiments across multilingual (Flickr30k-CN, CVLUE, XM3600), long-form (Wikipedia), and multimodal (CIRR) benchmarks demonstrate that GRAPE consistently improves performance, achieving an average gain of 4.9% in Recall@10 without any modification to the underlying retriever. The code is available at https://github.com/mogulzhang/GRAPE.