mlsanityData to 14 Oct 2026

Mode-Dependent Rectification for Stable PPO Training

arXiv 2602.05619 ↗First posted 5 Feb 2026cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout) are commonly used in visual reinforcement learning but can destabilize on-policy optimization. We show that in Proximal Policy Optimization (PPO), discrepancies between training and evaluation behavior induced by Batch Normalization lead to policy mismatch, distributional drift, and reward collapse. We propose Mode-Dependent Rectification (MDR), a lightweight dual-phase training procedure that stabilizes PPO under mode-dependent layers without architectural changes. Experiments across procedurally generated games and real-world patch-localization tasks demonstrate that MDR consistently improves stability and performance, and extends naturally to other mode-dependent layers.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 36 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

26 of 36 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2018Group Normalizationcited by 1,524
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
14 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  6. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  7. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  8. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  9. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  10. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  11. 2021CrossNorm and SelfNorm for Generalization under Distribution Shiftscited by 33
  12. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  13. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  14. 2019Rogue-Gym: A New Challenge for Generalization in Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Interactive Whole Slide Images for RL-based Tumour Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  2. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
  3. 2026Not All Transitions Matter: Evidence from PPO
  4. 2024Beyond the Boundaries of Proximal Policy Optimization
  5. 2026EasyPPO: Stabilizing the Critic Is Key
  6. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip
  7. 2026TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
  8. 2026ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm