mlsanityData to 7 Oct 2026

Is High Variance Unavoidable in RL? A Case Study in Continuous Control

arXiv 2110.11222 ↗First posted 21 Oct 2021cs.LG cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle for real-world applications, where safety and predictability are paramount. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance -- continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that variance mostly arises early in training as a result of poor "outlier" runs, but that weight initialization and initial exploration are not to blame. We show that one cause for early variance is numerical instability which leads to saturating nonlinearities. We investigate several fixes to this issue and find that one particular method is surprisingly effective and simple -- normalizing penultimate features. Addressing the learning instability allows for larger learning rates, and significantly decreases the variance of outcomes. This demonstrates that the perceived variance in RL is not necessarily inherent to the problem definition and may be addressed through simple architectural modifications.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

43 of 63 references.

References · 43

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
31 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  4. 2017Deep Reinforcement Learning that Matterscited by 727
  5. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2017Convolutional Sequence to Sequence Learningcited by 550
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  12. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  13. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  14. 2018Optimization of Molecules via Deep Reinforcement Learningcited by 199
  15. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  16. 2018Understanding Batch Normalizationcited by 157
  17. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  18. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  19. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  20. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  21. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56
  22. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 49
  23. 2019On the adequacy of untuned warmup for adaptive optimizationcited by 35
  24. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 35
  25. 2018Variance Reduction for Reinforcement Learning in Input-Driven Environmentscited by 30
  26. 2019A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithmscited by 26
  27. 2019A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robotscited by 12
  28. 2019Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environmentscited by 9
  29. 2020Deep Reinforcement Learning-Based Beam Tracking for Low-Latency Services in Vehicular Networkscited by 4
  30. 2021Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precisioncited by 3
  31. 2020Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradientcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 7; 2025: 9; 2026: 5.

26 papers.

Sort:

By area:

  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  3. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  4. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  5. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  6. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  7. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  8. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  9. 2022The Impact of Task Underspecification in Evaluating Deep Reinforcement Learningcited by 10
  10. 2023DexCatch: Learning to Catch Arbitrary Objects with Dexterous Handscited by 8
  11. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 6
  12. 2024Identifying Policy Gradient Subspacescited by 5
14 more
  1. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  2. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  3. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  4. 2025KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillationcited by 2
  5. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  6. 2026Performance Variation in Deep Reinforcement Learningcited by 2
  7. 2024Hadamard Representation: Scaffolding Performance Across Model-free RLcited by 1
  8. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  9. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  10. 2024Actor-Critic Reinforcement Learning with Phased Actorcited by 0
  11. 2025Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Gamescited by 0
  12. 2026Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigationcited by 0
  13. 2026NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learningcited by 0
  14. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0