# The Mirage of Action-Dependent Baselines in Reinforcement Learning

arXiv: https://arxiv.org/abs/1802.10031 · HTML: https://mlsanity.com/paper/1802.10031/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2018.

## Built on by (most cited first, 30 of 48)

- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [Settling the Variance of Multi-Agent Policy Gradients](https://mlsanity.com/paper/2108.08612/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Vanishing Gradients in Reinforcement Finetuning of Language Models](https://mlsanity.com/paper/2310.20703/) (2023)
- [The Role of Baselines in Policy Gradient Optimization](https://mlsanity.com/paper/2301.06276/) (2023)
- [Causal Policy Gradient for Whole-Body Mobile Manipulation](https://mlsanity.com/paper/2305.04866/) (2023)
- [Understanding the Effect of Stochasticity in Policy Optimization](https://mlsanity.com/paper/2110.15572/) (2021)
- [Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.12030/) (2022)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [Position: Benchmarking is Limited in Reinforcement Learning Research](https://mlsanity.com/paper/2406.16241/) (2024)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [A unified view of likelihood ratio and reparameterization gradients](https://mlsanity.com/paper/2105.14900/) (2021)
- [Stochastic Gradient Succeeds for Bandits](https://mlsanity.com/paper/2402.17235/) (2024)
- [PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference](https://mlsanity.com/paper/2407.05010/) (2024)
- [Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs](https://mlsanity.com/paper/2102.10362/) (2021)
- [Search in Imperfect Information Games](https://mlsanity.com/paper/2111.05884/) (2021)
- [BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents](https://mlsanity.com/paper/2606.25556/) (2026)
- [Improving and Benchmarking Offline Reinforcement Learning Algorithms](https://mlsanity.com/paper/2306.00972/) (2023)
- [Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation](https://mlsanity.com/paper/2106.13125/) (2021)
- [An Invitation to Deep Reinforcement Learning](https://mlsanity.com/paper/2312.08365/) (2023)
- [Improving Value Estimation Critically Enhances Vanilla Policy Gradient](https://mlsanity.com/paper/2505.19247/) (2025)
- [Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States](https://mlsanity.com/paper/2605.07579/) (2026)
- [Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization](https://mlsanity.com/paper/2206.06827/) (2022)
- [A reinforcement learning approach to resource allocation in genomic selection](https://mlsanity.com/paper/2107.10901/) (2021)
- [Evaluating and Enhancing Large Language Models for Conversational Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2312.11282/) (2023)
- [Interpretable and Effective Reinforcement Learning for Attacking against Graph-based Rumor Detection](https://mlsanity.com/paper/2201.05819/) (2022)

## Abstract (arXiv)

Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to depend on both the state and action and suggest that this significantly reduces variance and improves sample efficiency without introducing bias into the gradient estimates. To better understand this development, we decompose the variance of the policy gradient estimator and numerically show that learned state-action-dependent baselines do not in fact reduce variance over a state-dependent baseline in commonly tested benchmark domains. We confirm this unexpected result by reviewing the open-source code accompanying these prior papers, and show that subtle implementation decisions cause deviations from the methods presented in the papers and explain the source of the previously observed empirical gains. Furthermore, the variance decomposition highlights areas for improvement, which we demonstrate by illustrating a simple change to the typical value function parameterization that can significantly improve performance.