mlsanityData to 14 Oct 2026

Matrix Low-Rank Approximation For Policy Gradient Methods

arXiv 2405.17626 ↗First posted 27 May 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Estimating a policy that maps states to actions is a central problem in reinforcement learning. Traditionally, policies are inferred from the so called value functions (VFs), but exact VF computation suffers from the curse of dimensionality. Policy gradient (PG) methods bypass this by learning directly a parametric stochastic policy. Typically, the parameters of the policy are estimated using neural networks (NNs) tuned via stochastic gradient descent. However, finding adequate NN architectures can be challenging, and convergence issues are common as well. In this paper, we put forth low-rank matrix-based models to estimate efficiently the parameters of PG algorithms. We collect the parameters of the stochastic policy into a matrix, and then, we leverage matrix-completion techniques to promote (enforce) low rank. We demonstrate via numerical studies how low-rank matrix-based policy models reduce the computational and sample complexities relative to NN models, while achieving a similar aggregated reward.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 31 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

14 of 31 references.

References · 14

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2016OpenAI Gymcited by 1,736
  3. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  4. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  5. 2014Generalized Low Rank Modelscited by 55
  6. 2020Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimationcited by 25
  7. 2021Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learningcited by 25
  8. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  9. 2019Optimization for Reinforcement Learning: From Single Agent to Cooperative Agentscited by 20
  10. 2017Expected Policy Gradientscited by 16
  11. 2022Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learningcited by 16
  12. 2021Low-rank State-action Value-function Approximationcited by 10
2 more
  1. 2015Value function approximation via low-rank modelscited by 7
  2. 2018Nonparametric Stochastic Compositional Gradient Descent for Q-Learning in Continuous Markov Decision Problemscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learningcited by 3
  2. 2025A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Matrix Low-Rank Trust Region Policy Optimization
  2. 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation
  3. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradients
  4. 2024Identifying Policy Gradient Subspaces
  5. 2023Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure
  6. 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximation
  7. 2026Bayesian policy gradient and actor-critic algorithms
  8. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations