mlsanityData to 7 Oct 2026

Batch size-invariance for policy optimization

arXiv 2110.00641 ↗First posted 1 Oct 2021cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

22 of 34 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
10 more
  1. 2018An Empirical Model of Large-Batch Trainingcited by 268
  2. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  3. 2017A unified view of entropy-regularized Markov decision processescited by 174
  4. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  6. 2020Phasic Policy Gradientcited by 121
  7. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  8. 2020Mirror Descent Policy Optimizationcited by 102
  9. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 35
  10. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 4; 2025: 9; 2026: 14.

33 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  3. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  4. 2023Scaling laws for single-agent reinforcement learningcited by 39
  5. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 38
  6. 2023Small batch deep reinforcement learningcited by 35
  7. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  8. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  9. 2022VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangementcited by 18
  10. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 13
  11. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  12. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 11
21 more
  1. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 9
  2. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 8
  3. 2025Truncated Proximal Policy Optimizationcited by 8
  4. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  5. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 6
  6. 2025Optimal Scaling Needs Optimal Normcited by 5
  7. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatchcited by 5
  8. 2024Batch size invariant Adamcited by 4
  9. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 4
  10. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  11. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  12. 2022Relative Sparsity for Medical Decision Problemscited by 2
  13. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2
  14. 2023Guaranteed Trust Region Optimization via Two-Phase KL Penalizationcited by 1
  15. 2026When Do Larger Batches Help Scale LLM Reinforcement Learning?cited by 1
  16. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Trainingcited by 0
  17. 2026Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagationcited by 0
  18. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  19. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 0
  20. 2026Re3Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learningcited by 0
  21. 2026Validity-Preserving Hierarchical RL for Joint Routing and Switch Placement in EDAcited by 0