mlsanityData to 14 Oct 2026

Adaptive Policy Synchronization for Scalable Reinforcement Learning

arXiv 2507.10990 ↗First posted 15 Jul 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Scaling reinforcement learning (RL) often requires running environments across many machines, but most frameworks tie simulation, training, and infrastructure into rigid systems. We introduce ClusterEnv, a lightweight interface for distributed environment execution that preserves the familiar Gymnasium API. ClusterEnv uses the DETACH pattern, which moves environment reset() and step() operations to remote workers while keeping learning centralized. To reduce policy staleness without heavy communication, we propose Adaptive Policy Synchronization (APS), where workers request updates only when divergence from the central learner grows too large. ClusterEnv supports both on- and off-policy methods, integrates into existing training code with minimal changes, and runs efficiently on clusters. Experiments on discrete control tasks show that APS maintains performance while cutting synchronization overhead. Source code is available at https://github.com/rodlaf/ClusterEnv.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 15 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

12 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey8 Nov

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  7. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  8. 2018Distributed Prioritized Experience Replaycited by 269
  9. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  10. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  11. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Scalable Reinforcement Learning via Adaptive Batch Scaling
  2. 2026SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
  3. 2025Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
  4. 2025Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning
  5. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning
  6. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
  7. 2025Harnessing Data from Clustered LQR Systems: Personalized and Collaborative Policy Optimization
  8. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing