mlsanityData to 7 Oct 2026

Identifying Policy Gradient Subspaces

arXiv 2401.06604 ↗First posted 12 Jan 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

30 of 46 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  7. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2017Parameter Space Noise for Explorationcited by 158
18 more
  1. 2023Extreme Parkour with Legged Robotscited by 120
  2. 2018Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learningcited by 101
  3. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  5. 2021Subspace Adversarial Trainingcited by 41
  6. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  7. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  8. 2022DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systemscited by 27
  9. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 26
  10. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  11. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  12. 2013Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methodscited by 9
  13. 2022Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environmentscited by 9
  14. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8
  15. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7
  16. 2020Quasi-Newton's method in the class gradient defined high-curvature subspacecited by 3
  17. 2021Unified Data Collection for Visual-Inertial Calibration via Deep Reinforcement Learningcited by 3
  18. 2023The Ladder in Chaos: A Simple and Effective Improvement to General DRL Algorithms by Policy Path Trimming and Boostingcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  3. 2025Low-Rank Agent-Specific Adaptation (LoRASA) for Multi-Agent Policy Learningcited by 4
  4. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  5. 2025Can We Optimize Deep RL Policy Weights as Trajectory Modeling?cited by 0