mlsanityData to 7 Oct 2026

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

arXiv 2201.09736 ↗First posted 21 Jan 2022cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 21 of its 56 references that are in the map. First posted on arXiv 21 Jan 2022.

Builds on

21 of 56 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2016OpenAI Gymcited by 1,684
  4. 2015Prioritized Experience Replaycited by 1,070
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  6. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  7. 2016Tensor Decomposition for Signal Processing and Machine Learningcited by 223
  8. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  9. 2016TensorLy: Tensor Learning in Pythoncited by 104
  10. 2018Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractionscited by 84
  11. 2014Generalized Low Rank Modelscited by 55
  12. 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 51
9 more
  1. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 28
  2. 2020Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimationcited by 25
  3. 2021Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learningcited by 25
  4. 2019Harnessing Structures for Value-Based Planning and Reinforcement Learningcited by 24
  5. 2019Hybrid Reinforcement Learning with Expert State Sequencescited by 14
  6. 2016Reinforcement Learning in Rich-Observation MDPs using Spectral Methodscited by 13
  7. 2021Low-rank State-action Value-function Approximationcited by 10
  8. 2015Value function approximation via low-rank modelscited by 7
  9. 2021Model based Multi-agent Reinforcement Learning with Tensor Decompositionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 6; 2025: 7; 2026: 2.

16 papers.

Sort:

  1. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  2. 2022Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structurecited by 7
  3. 2024Tensor Low-rank Approximation of Finite-horizon Value Functionscited by 3
  4. 2025Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learningcited by 3
  5. 2024Matrix Low-Rank Trust Region Policy Optimizationcited by 2
  6. 2024Matrix Low-Rank Approximation For Policy Gradient Methodscited by 2
  7. 2024Low-Rank Tensors for Multi-Dimensional Markov Modelscited by 2
  8. 2024The Limits of Transfer Reinforcement Learning with Latent Low-rank Structurecited by 1
  9. 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimationcited by 1
  10. 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximationcited by 1
  11. 2025A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learningcited by 0
  12. 2025Unrolling Dynamic Programming via Graph Filterscited by 0
4 more
  1. 2025ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertaintycited by 0
  2. 2025Tensor-Efficient High-Dimensional Q-learningcited by 0
  3. 2026Factorized Spectral Representations for Reinforcement Learningcited by 0
  4. 2026A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learningcited by 0