GPG: Generalized Policy Gradient Theorem for Transformer-based Policies
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies. Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework. Furthermore, we explore its practical applications in training Large Language Models (LLMs), offering new insights into efficient policy optimization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 38 references that are in the map. First posted on arXiv 11 Dec 2025.
Builds on
29 of 38 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2023Let's Verify Step by Stepcited by 3,789
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2015Trust Region Policy Optimizationcited by 2,580
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
17 more
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
- 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
- 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
- 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
- 2025Agentic Reinforced Policy Optimizationcited by 172
- 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learningcited by 72
- 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
- 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
- 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
- 2024PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistencycited by 27
- 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAGcited by 27
- 2025DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answeringcited by 20
- 2024SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Modelscited by 17
- 2024SQL-to-Schema Enhances Schema Linking in Text-to-SQLcited by 12
- 2025General Table Question Answering via Answer-Formula Joint Generationcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
- 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
- 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
- 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
- 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
- 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
- 2025Geometric-Mean Policy Optimization
- 2026Holder Policy Optimisation