mlsanityData to 7 Oct 2026

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

arXiv 2306.01460 ↗First posted 2 Jun 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper proposes a step toward approximate Bayesian inference in on-policy actor-critic deep reinforcement learning. It is implemented through three changes to the Asynchronous Advantage Actor-Critic (A3C) algorithm: (1) applying a ReLU function to advantage estimates, (2) spectral normalization of actor-critic weights, and (3) incorporating dropout as a Bayesian approximation. We prove under standard assumptions that restricting policy updates to positive advantages optimizes for value by maximizing a lower bound on the value function plus an additive term. We show that the additive term is bounded proportional to the Lipschitz constant of the value function, which offers theoretical grounding for spectral normalization of critic weights. Finally, our application of dropout corresponds to approximate Bayesian inference over both the actor and critic parameters, which enables textitadaptive state-aware exploration around the modes of the actor via Thompson sampling. We demonstrate significant improvements for median and interquartile mean metrics over A3C, PPO, SAC, and TD3 on the MuJoCo continuous control benchmark and improvement over PPO in the challenging ProcGen generalization benchmark.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 54 of its 97 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

54 of 97 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2016Layer Normalizationcited by 5,579
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
38 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2016OpenAI Gymcited by 1,684
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  10. 2018Learning Dexterous In-Hand Manipulationcited by 643
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
  13. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  14. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  15. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  16. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  17. 2019Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientscited by 294
  18. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  19. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  20. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  21. 2018Self-Imitation Learningcited by 135
  22. 2022Discovered Policy Optimisationcited by 103
  23. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  24. 2014Optimizing the CVaR via Samplingcited by 98
  25. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  26. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  27. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  28. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  29. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  30. 2019Worst Cases Policy Gradientscited by 52
  31. 2022Exploration via Elliptical Episodic Bonusescited by 51
  32. 2019AlphaStar: An Evolutionary Computation Perspectivecited by 50
  33. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  34. 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
  35. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  36. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 23
  37. 2017Efficient exploration with Double Uncertain Value Networkscited by 15
  38. 2020Self-Imitation Advantage Learningcited by 15

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 2.

10 papers.

  1. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  2. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  3. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scalecited by 6
  4. 2025The Resurrection of the ReLUcited by 5
  5. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Studycited by 5
  6. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  7. 2024Stein Variational Evolution Strategiescited by 2
  8. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  9. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  10. 2026What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floorcited by 0