mlsanityData to 14 Oct 2026

Matrix Low-Rank Trust Region Policy Optimization

arXiv 2405.17625 ↗First posted 27 May 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Most methods in reinforcement learning use a Policy Gradient (PG) approach to learn a parametric stochastic policy that maps states to actions. The standard approach is to implement such a mapping via a neural network (NN) whose parameters are optimized using stochastic gradient descent. However, PG methods are prone to large policy updates that can render learning inefficient. Trust region algorithms, like Trust Region Policy Optimization (TRPO), constrain the policy update step, ensuring monotonic improvements. This paper introduces low-rank matrix-based models as an efficient alternative for estimating the parameters of TRPO algorithms. By gathering the stochastic policy's parameters into a matrix and applying matrix-completion techniques, we promote and enforce low rank. Our numerical studies demonstrate that low-rank matrix-based policy models effectively reduce both computational and sample complexities compared to NN models, while maintaining comparable aggregated rewards.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 31 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

15 of 31 references.

References · 15

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016OpenAI Gymcited by 1,736
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  6. 2014Generalized Low Rank Modelscited by 55
  7. 2020Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimationcited by 25
  8. 2021Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learningcited by 25
  9. 2018Factorized Q-Learning for Large-Scale Multi-Agent Systemscited by 19
  10. 2017Expected Policy Gradientscited by 16
  11. 2022Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learningcited by 16
  12. 2021Low-rank State-action Value-function Approximationcited by 10
3 more
  1. 2015Value function approximation via low-rank modelscited by 7
  2. 2022Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structurecited by 7
  3. 2018Nonparametric Stochastic Compositional Gradient Descent for Q-Learning in Continuous Markov Decision Problemscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learningcited by 3
  2. 2026Ratio-Variance Regularized Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Matrix Low-Rank Approximation For Policy Gradient Methods
  2. 2024Simple Policy Optimization
  3. 2024Embedding Safety into RL: A New Take on Trust Region Methods
  4. 2018Model-Ensemble Trust-Region Policy Optimization
  5. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
  6. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
  7. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
  8. 2023Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure