# Smoothed Action Value Functions for Learning Gaussian Policies

arXiv: https://arxiv.org/abs/1803.02348 · HTML: https://mlsanity.com/paper/1803.02348/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2018.

## Built on by (most cited first, 7 of 7)

- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Rethinking Population-assisted Off-policy Reinforcement Learning](https://mlsanity.com/paper/2305.02949/) (2023)
- [STL-Based Synthesis of Feedback Controllers Using Reinforcement Learning](https://mlsanity.com/paper/2212.01022/) (2022)
- [Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios](https://mlsanity.com/paper/2412.06390/) (2024)
- [Control Synthesis of Cyber-Physical Systems for Real-Time Specifications through Causation-Guided Reinforcement Learning](https://mlsanity.com/paper/2510.07715/) (2025)
- [Soft Deterministic Policy Gradient with Gaussian Smoothing](https://mlsanity.com/paper/2605.06228/) (2026)
- [FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion](https://mlsanity.com/paper/2606.31691/) (2026)

## Abstract (arXiv)

State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value. We show that such smoothed Q-values still satisfy a Bellman equation, making them learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships, we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator. The approach is additionally amenable to proximal optimization by augmenting the objective with a penalty on KL-divergence from a previous policy. We find that the ability to learn both a mean and covariance during training leads to significantly improved results on standard continuous control benchmarks.