mlsanityData to 7 Oct 2026

Stabilizing Off-Policy Deep Reinforcement Learning from Pixels

arXiv 2207.00986 ↗First posted 3 Jul 2022cs.LG cs.AI cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Off-policy reinforcement learning (RL) from pixel observations is notoriously unstable. As a result, many successful algorithms must combine different domain-specific practices and auxiliary losses to learn meaningful behaviors in complex environments. In this work, we provide novel analysis demonstrating that these instabilities arise from performing temporal-difference learning with a convolutional encoder and low-magnitude rewards. We show that this new visual deadly triad causes unstable training and premature convergence to degenerate solutions, a phenomenon we name catastrophic self-overfitting. Based on our analysis, we propose A-LIX, a method providing adaptive regularization to the encoder's gradients that explicitly prevents the occurrence of catastrophic self-overfitting using a dual objective. By applying A-LIX, we significantly outperform the prior state-of-the-art on the DeepMind Control and Atari 100k benchmarks without any data augmentation or auxiliary losses.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 3 Jul 2022.

Builds on

56 of 78 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2021Convex optimizationcited by 2,479
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
38 more
  1. 2018On the Spectral Bias of Neural Networkscited by 1,321
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2018DeepMind Control Suitecited by 935
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  13. 2017Exploring Generalization in Deep Learningcited by 595
  14. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  15. 2019Model-Based Reinforcement Learning for Ataricited by 495
  16. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  17. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2019Challenges of Real-World Reinforcement Learningcited by 282
  20. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  21. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  22. 2018DropBlock: A regularization method for convolutional networkscited by 261
  23. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  24. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  25. 2021Offline RL Without Off-Policy Evaluationcited by 150
  26. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  27. 2020Phasic Policy Gradientcited by 121
  28. 2019When to use parametric models in reinforcement learning?cited by 118
  29. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  30. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  31. 2019Observational Overfitting in Reinforcement Learningcited by 90
  32. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  33. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  34. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 58
  35. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  36. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  37. 2020A case for new neural network smoothness constraintscited by 43
  38. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 8; 2025: 1; 2026: 6.

32 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  3. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  4. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  5. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  6. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  7. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  8. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  9. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  10. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  11. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  12. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
20 more
  1. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 20
  2. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 16
  3. 2024Reinforcement Learning from Delayed Observations via World Modelscited by 13
  4. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  5. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  6. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  7. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  8. 2023On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learningcited by 8
  9. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  10. 2023An Invitation to Deep Reinforcement Learningcited by 5
  11. 2023RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environmentscited by 4
  12. 2024Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulationcited by 4
  13. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  14. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  15. 2023Active Vision Reinforcement Learning under Limited Visual Observabilitycited by 0
  16. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  17. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learningcited by 0
  18. 2026Autonomous Diffractometry Enabled by Visual Reinforcement Learningcited by 0
  19. 2026Mask-based Predictive Representations for Reinforcement Learningcited by 0
  20. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0