mlsanityData to 7 Oct 2026

The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

arXiv 1301.2315 ↗First posted 10 Jan 2013cs.LG cs.AI stat.MLcited by 84

Not on the map. Why?

Abstract (arXiv)

There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator hasbeen found to be a significant practical problem. Recent approacheshave discounted future rewards, introducing a bias-variance trade-offinto the gradient estimate. We incorporate a reward baseline into thelearning system, and show that it affects variance without introducingfurther bias. In particular, as we approach the zero-bias,high-variance parameterization, the optimal (or variance minimizing)constant reward baseline is equal to the long-term average expectedreward. Modified policy-gradient algorithms are presented, and anumber of experiments demonstrate their improvement over previous work.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jan 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 18; 2023: 15; 2024: 9; 2025: 12; 2026: 15.

84 papers.

Sort:

By area:

  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  2. 2022GFlowNets and variational inferencecited by 103
  3. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  4. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  5. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  6. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  7. 2025On-Policy RL with Optimal Reward Baselinecited by 30
  8. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  9. 2022Gradient Estimation with Discrete Stein Operatorscited by 22
  10. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
  11. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  12. 2022Asynchronous Actor-Critic for Multi-Agent Reinforcement Learningcited by 16
38 more
  1. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  2. 2024QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCEcited by 13
  3. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 13
  4. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  5. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
  6. 2023System-status-aware Adaptive Network for Online Streaming Video Understandingcited by 9
  7. 2023IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARLcited by 9
  8. 2023Simultaneous Machine Translation with Tailored Referencecited by 9
  9. 2024Policy Gradient with Active Importance Samplingcited by 9
  10. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 9
  11. 2021Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPscited by 8
  12. 2021Double Control Variates for Gradient Estimation in Discrete Latent Variable Modelscited by 8
  13. 2025Aligning Constraint Generation with Design Intent in Parametric CADcited by 8
  14. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  15. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  16. 2023Dynamic Feature-based Deep Reinforcement Learning for Flow Control of Circular Cylinder with Sparse Surface Pressure Sensingcited by 7
  17. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 7
  18. 2021Policy Gradients Incorporating the Futurecited by 6
  19. 2022DeepCreativity: Measuring Creativity with Deep Learning Techniquescited by 6
  20. 2024On Divergence Measures for Training GFlowNetscited by 6
  21. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  22. 2023GFN-SR: Symbolic Regression with Generative Flow Networkscited by 5
  23. 2023An Invitation to Deep Reinforcement Learningcited by 5
  24. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5
  25. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RLcited by 5
  26. 2021End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimizationcited by 4
  27. 2022Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimizationcited by 4
  28. 2021Diversity-Driven Combination for Grammatical Error Correctioncited by 3
  29. 2021Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channelcited by 3
  30. 2022An Unpooling Layer for Graph Generationcited by 3
  31. 2022Cooperation and Competition: Flocking with Evolutionary Multi-Agent Reinforcement Learningcited by 3
  32. 2022Rephrasing the Reference for Non-Autoregressive Machine Translationcited by 3
  33. 2023Policy Gradient with Tree Expansioncited by 3
  34. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
  35. 2024Differentiable Discrete Event Simulation for Queuing Network Controlcited by 3
  36. 2022Interpretable and Effective Reinforcement Learning for Attacking against Graph-based Rumor Detectioncited by 2
  37. 2022Policy Learning and Evaluation with Randomized Quasi-Monte Carlocited by 2
  38. 2022Data-driven End-to-end Learning of Pole Placement Control for Nonlinear Dynamics via Koopman Invariant Subspacescited by 2

50 most cited of 84 All 84 (JSON)