mlsanityData to 14 Oct 2026

Composing Policy Gradients and Prompt Optimization for Language Model Programs

arXiv 2508.04660 ↗First posted 6 Aug 2025cs.CLcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has proven to be an effective tool for post-training language models (LMs). However, AI systems are increasingly expressed as modular programs that mix together multiple LM calls with distinct prompt templates and other tools, and it is not clear how practitioners can best leverage online RL algorithms like GRPO to improve these systems. We begin to address this challenge by investigating whether it is possible to effectively instantiate GRPO for arbitrary multi-prompt programs and whether it can work robustly as an off-the-shelf optimizer for LM programs using the same abstractions and constraints typically involved for prompt optimization. Our main variant of multi-module GRPO constructs groups from module-level invocations, and we also consider trajectory-level grouping as another natural instantiation. We find for the first time that GRPO (and its multi-module counterpart) empirically composes well with automatic prompt optimization, and together they improve accuracy by 11% on average across classification, many-hop search, and privacy-preserving delegation tasks against the post-trained LM - with 5% gains against prompt optimization on its own. We open-source multi-module GRPO in the DSPy library at https://dspy.ai .

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 43 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

22 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  9. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  12. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
7 more
  1. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  2. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  3. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  4. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  5. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 57
  6. 2020Answering Open-Domain Questions of Varying Reasoning Steps from Textcited by 43
  7. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026WHALE: A Simple Recipe for Joint Harness-Weight Optimizationcited by 11
  2. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoningcited by 1
  3. 2025WRAVAL -- WRiting Assist eVALuationcited by 0
  4. 2026From Agent Failures to Text Policies: What Works and What Breakscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  3. 2026GRPO Training Dynamics for Small Language Models
  4. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  5. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  6. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  7. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  8. 2026Holder Policy Optimisation