mlsanityData to 7 Oct 2026

SAPG: Split and Aggregate Policy Gradients

arXiv 2407.20230 ↗First posted 29 Jul 2024cs.LG cs.AI cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

Despite extreme sample inefficiency, on-policy reinforcement learning, aka policy gradients, has become a fundamental tool in decision-making problems. With the recent advances in GPU-driven simulation, the ability to collect large amounts of data for RL training has scaled exponentially. However, we show that current RL methods, e.g. PPO, fail to ingest the benefit of parallelized environments beyond a certain point and their performance saturates. To address this, we propose a new on-policy RL algorithm that can effectively leverage large-scale environments by splitting them into chunks and fusing them back together via importance sampling. Our algorithm, termed SAPG, shows significantly higher performance across a variety of challenging environments where vanilla PPO and other strong baselines fail to achieve high performance. Website at https://sapg-rl.github.io/

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

23 of 32 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  9. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
11 more
  1. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  2. 2018Distributed Prioritized Experience Replaycited by 260
  3. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  4. 2021A System for General In-Hand Object Re-Orientationcited by 155
  5. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  6. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  7. 2023Extreme Parkour with Legged Robotscited by 120
  8. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  9. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  10. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  11. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2025Horizon Reduction Makes RL Scalablecited by 55
  3. 2025Value-Based Deep RL Scales Predictablycited by 14
  4. 2025LocoFormer: Generalist Locomotion via Long-context Adaptationcited by 14
  5. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  6. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  7. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 10
  8. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  9. 2025Evolutionary Policy Optimizationcited by 3
  10. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  11. 2025Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learningcited by 1
  12. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
5 more
  1. 2026Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?cited by 1
  2. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  3. 2026Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learningcited by 0
  4. 2026Remember to be Curious: Episodic Context and Persistent Worlds for 3D Explorationcited by 0
  5. 2026Not All Transitions Matter: Evidence from PPOcited by 0