mlsanityData to 14 Oct 2026

Taming the Noise in Reinforcement Learning via Soft Updates

arXiv 1512.08562 ↗First posted 28 Dec 2015cs.LG cs.IT math.ITcited by 127

Not on the map. Why?

Cited by 38 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Model-free reinforcement learning algorithms, such as Q-learning, perform poorly in the early stages of learning in noisy environments, because much effort is spent unlearning biased estimates of the state-action value function. The bias results from selecting, among several noisy estimates, the apparent optimum, which may actually be suboptimal. We propose G-learning, a new off-policy learning algorithm that regularizes the value estimates by penalizing deterministic policies in the beginning of the learning process. We show that this method reduces the bias of the value-function estimation, leading to faster convergence to the optimal value and the optimal policy. Moreover, G-learning enables the natural incorporation of prior domain knowledge, when available. The stochastic nature of G-learning also makes it avoid some exploration costs, a property usually attributed only to on-policy algorithms. We illustrate these ideas in several examples, where G-learning results in significant improvements of the convergence rate and the cost of the learning process.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 36; 2022: 16; 2023: 18; 2024: 26; 2025: 16; 2026: 13.

127 papers.

Sort:

By area:

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  6. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  9. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  10. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  11. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  12. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
38 more
  1. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  2. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  3. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  4. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
  5. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  6. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  7. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  8. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  9. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  10. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  11. 2023Coherent Soft Imitation Learningcited by 22
  12. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  13. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  14. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  15. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 20
  16. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  17. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  18. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  19. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  20. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  21. 2021Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learningcited by 15
  22. 2024The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithmcited by 15
  23. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  24. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matchingcited by 14
  25. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  26. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  27. 2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
  28. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  29. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  30. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  31. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  32. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Surveycited by 11
  33. 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10
  34. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 10
  35. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  36. 2023Time-series Generation by Contrastive Imitationcited by 9
  37. 2021Learning Altruistic Behaviours in Reinforcement Learning without External Rewardscited by 8
  38. 2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8

50 most cited of 127 All 127 (JSON)