mlsanityData to 7 Oct 2026

Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies

arXiv 2502.20190 ↗First posted 27 Feb 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

As the demands for superior agents grow, the training complexity of Deep Reinforcement Learning (DRL) becomes higher. Thus, accelerating training of DRL has become a major research focus. Dividing the DRL training process into subtasks and using parallel computation can effectively reduce training costs. However, current DRL training systems lack sufficient parallelization due to data assignment between subtask components. This assignment issue has been ignored, but addressing it can further boost training efficiency. Therefore, we propose a high-throughput distributed RL training system called TianJi. It relaxes assignment dependencies between subtask components and enables event-driven asynchronous communication. Meanwhile, TianJi maintains clear boundaries between subtask components. To address convergence uncertainty from relaxed assignment dependencies, TianJi proposes a distributed strategy based on the balance of sample production and consumption. The strategy controls the staleness of samples to correct their quality, ensuring convergence. We conducted extensive experiments. TianJi achieves a convergence time acceleration ratio of up to 4.37 compared to related comparison systems. When scaled to eight computational nodes, TianJi shows a convergence time speedup of 1.6 and a throughput speedup of 7.13 relative to XingTian, demonstrating its capability to accelerate training and scalability. In data transmission efficiency experiments, TianJi significantly outperforms other systems, approaching hardware limits. TianJi also shows effectiveness in on-policy algorithms, achieving convergence time acceleration ratios of 4.36 and 2.95 compared to RLlib and XingTian. TianJi is accessible at https://github.com/HiPRL/TianJi.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

27 of 45 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2017Deep Reinforcement Learning: An Overviewcited by 327
  10. 2018Distributed Prioritized Experience Replaycited by 260
  11. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  12. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
15 more
  1. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  2. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  3. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  4. 2021Reverb: A Framework For Experience Replaycited by 27
  5. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 23
  6. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  7. 2019Gossip-based Actor-Learner Architectures for Deep Reinforcement Learningcited by 16
  8. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  9. 2023SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Corescited by 10
  10. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  11. 2022DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matchingcited by 5
  12. 2022MSRL: Distributed Reinforcement Learning with Dataflow Fragmentscited by 4
  13. 2023Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networkscited by 4
  14. 2024Rocket Landing Control with Random Annealing Jump Start Reinforcement Learningcited by 3
  15. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimizationcited by 1

Built on by

Not cited yet.