mlsanityData to 14 Oct 2026

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning

arXiv 2403.07704 ↗First posted 12 Mar 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gaussian error distribution. However, a recent study suggested that the error distribution for training the value function is often skewed because of the properties of the Bellman operator, and violates the implicit assumption of normal error distribution in the least squares method. To address this, we proposed a method called Symmetric Q-learning, in which the synthetic noise generated from a zero-mean distribution is added to the target values to generate a Gaussian error distribution. We evaluated the proposed method on continuous control benchmark tasks in MuJoCo. It improved the sample efficiency of a state-of-the-art reinforcement learning method by reducing the skewness of the error distribution.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 10 of its 24 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

10 of 24 references.

References · 10

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  10. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Deep Weighted Bellman Residual Minimization for Q^* Estimation
  2. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
  3. 2023LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning
  5. 2024Stabilizing Extreme Q-learning by Maclaurin Expansion
  6. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
  7. 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
  8. 2026Residuals-based Offline Reinforcement Learning