mlsanityData to 14 Oct 2026

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks

arXiv 2506.03404 ↗First posted 3 Jun 2025cs.LG cs.AIcited by 13

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 67 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

40 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey8 Nov
  2. 2024Simplifying Deep Temporal Difference Learning5 Jul
  3. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  4. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  5. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning29 May
  6. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO1 May
  7. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  8. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  10. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  11. 2018Distributed Prioritized Experience Replaycited by 269
  12. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
20 more
  1. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  3. 2023Understanding plasticity in neural networkscited by 174
  4. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 151
  5. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  8. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  9. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  10. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 51
  11. 2021A study on the plasticity of neural networkscited by 44
  12. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  13. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  14. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  15. 2023Small batch deep reinforcement learningcited by 36
  16. 2021Batch size-invariance for policy optimizationcited by 35
  17. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  18. 2022The Phenomenon of Policy Churncited by 28
  19. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  20. 2024SAPG: Split and Aggregate Policy Gradientscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  3. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  4. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  5. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  6. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  7. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learnercited by 1
  8. 2026X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matchingcited by 1
  9. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  10. 2026Forager: a lightweight testbed for continual learning with partial observability in RLcited by 0
  11. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  12. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
1 more
  1. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story
  2. 2021A Dataset Perspective on Offline Reinforcement Learning
  3. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  4. 2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment
  5. 2025Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  7. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  8. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?