# Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines

arXiv: https://arxiv.org/abs/1706.06643 · HTML: https://mlsanity.com/paper/1706.06643/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2017.

## Built on by (most cited first, 20 of 20)

- [Risk-Sensitive Reinforcement Learning with Exponential Criteria](https://mlsanity.com/paper/2212.09010/) (2022)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [SEM: Reinforcement Learning for Search-Efficient Large Language Models](https://mlsanity.com/paper/2505.07903/) (2025)
- [Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs](https://mlsanity.com/paper/2102.10362/) (2021)
- [Doubly Optimal Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2410.02226/) (2024)
- [Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMs](https://mlsanity.com/paper/2505.12833/) (2025)
- [Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing](https://mlsanity.com/paper/2103.01009/) (2021)
- [Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings](https://mlsanity.com/paper/2111.00185/) (2021)
- [Meta-Reinforcement Learning for Reliable Communication in THz/VLC Wireless VR Networks](https://mlsanity.com/paper/2102.12277/) (2021)
- [Policy Gradient with Tree Expansion](https://mlsanity.com/paper/2301.13236/) (2023)
- [KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation](https://mlsanity.com/paper/2504.19024/) (2025)
- [Generative deep learning for decision making in gas networks](https://mlsanity.com/paper/2102.02125/) (2021)
- [Ensemble Consensus-based Representation Deep Reinforcement Learning for Hybrid FSO/RF Communication Systems](https://mlsanity.com/paper/2108.02551/) (2021)
- [SoftTreeMax: Policy Gradient with Tree Search](https://mlsanity.com/paper/2209.13966/) (2022)
- [Tackling Variabilities in Autonomous Driving](https://mlsanity.com/paper/2104.10415/) (2021)
- [On Solving a Stochastic Shortest-Path Markov Decision Process as Probabilistic Inference](https://mlsanity.com/paper/2109.05866/) (2021)
- [Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback](https://mlsanity.com/paper/2109.07054/) (2021)
- [OmniVLA-RL: A Vision-Language-Action Model with Spatial Understanding and Online RL](https://mlsanity.com/paper/2604.17706/) (2026)
- [Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training](https://mlsanity.com/paper/2606.21498/) (2026)
- [VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning](https://mlsanity.com/paper/2610.10782/) (2026)

## Abstract (arXiv)

We show how an action-dependent baseline can be used by the policy gradient theorem using function approximation, originally presented with action-independent baselines by (Sutton et al. 2000).