Data to 14 Oct 2026

Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

arXiv 1803.07246 ↗First posted 20 Mar 2018cs.LG cs.AI stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 16; 2023: 12; 2024: 6; 2025: 10; 2026: 14.

68 papers.

Sort:

By area:

2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
2022GFlowNets and variational inferencecited by 106
2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
2025BNPO: Beta Normalization Policy Optimizationcited by 28
2025Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewardscited by 27
2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
2022Hyperbolic Deep Reinforcement Learningcited by 25
2023The Role of Baselines in Policy Gradient Optimizationcited by 25

38 more

2023Causal Policy Gradient for Whole-Body Mobile Manipulationcited by 21
2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20
2022Deep Reinforcement Learning for Inverse Inorganic Materials Designcited by 19
2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
2024Transformer-based Model Predictive Control: Trajectory Optimization via Sequence Modelingcited by 16
2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
2022Neural Set Function Extensions: Learning with Discrete Functions in High Dimensionscited by 14
2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 13
2024Stochastic Gradient Succeeds for Banditscited by 10
2021Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPscited by 8
2023Global Rewards in Multi-Agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systemscited by 8
2021Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environmentcited by 7
2023Adaptive Instrument Design for Indirect Experimentscited by 6
2025Performative Policy Gradient: Optimality in Performative Reinforcement Learningcited by 6
2021Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluationcited by 5
2021Learning Cooperative Multi-Agent Policies with Partial Reward Decouplingcited by 5
2025FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learningcited by 5
2022Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimizationcited by 4
2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
2021Soft Actor-Critic With Integer Actionscited by 3
2022Deep Reinforcement Learning for Data-Driven Adaptive Scanning in Ptychographycited by 3
2022Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chainscited by 3
2023Policy Gradient with Tree Expansioncited by 3
2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
2025CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformationcited by 3
2022Interpretable and Effective Reinforcement Learning for Attacking against Graph-based Rumor Detectioncited by 2
2023Co-learning Planning and Control Policies Constrained by Differentiable Logic Specificationscited by 2
2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 2
2026On Advantage Estimates for Max@K Policy Gradientscited by 2
2026GFlowRL: Scaling Distribution-Matching RL to Large Language Modelscited by 2
2026SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agentscited by 2
2021Coordinate-wise Control Variates for Deep Policy Gradientscited by 1
2022Variance Reduction based Experience Replay for Policy Optimizationcited by 1
2022SoftTreeMax: Policy Gradient with Tree Searchcited by 1
2023Distillation Policy Optimizationcited by 1
2024Oracle-Efficient Reinforcement Learning for Max Value Ensemblescited by 1

50 most cited of 68 All 68 (JSON)