mlsanityData to 14 Oct 2026

Spectral Decomposition Representation for Reinforcement Learning

arXiv 2208.09515 ↗First posted 19 Aug 2022cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

Representation learning often plays a critical role in reinforcement learning by managing the curse of dimensionality. A representative class of algorithms exploits a spectral decomposition of the stochastic transition dynamics to construct representations that enjoy strong theoretical properties in an idealized setting. However, current spectral methods suffer from limited applicability because they are constructed for state-only aggregation and derived from a policy-dependent transition kernel, without considering the issue of exploration. To address these issues, we propose an alternative spectral method, Spectral Decomposition Representation (SPEDER), that extracts a state-action abstraction from the dynamics without inducing spurious dependence on the data collection policy, while also balancing the exploration-versus-exploitation trade-off during learning. A theoretical analysis establishes the sample efficiency of the proposed algorithm in both the online and offline settings. In addition, an experimental investigation demonstrates superior performance over current state-of-the-art algorithms across several benchmarks.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 66 of its 89 references that are in the map. First posted on arXiv 19 Aug 2022.

Builds on

66 of 89 references.

References · 66

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2020Mastering Atari with Discrete World Modelscited by 1,007
  9. 2018DeepMind Control Suitecited by 956
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
38 more
  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2018Is Q-learning Provably Efficient?cited by 487
  5. 2020Reinforcement Learning with Augmented Datacited by 468
  6. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  11. 2019Learning Latent Plans from Playcited by 316
  12. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  13. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  14. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  15. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  16. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  17. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  18. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  19. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  20. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  21. 2019Efficient Exploration via State Marginal Matchingcited by 188
  22. 2021Reinforcement Learning with Prototypical Representationscited by 179
  23. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  24. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  25. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  26. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  27. 2019Unsupervised State Representation Learning in Ataricited by 144
  28. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  29. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  30. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  31. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  32. 2017Value Prediction Networkcited by 117
  33. 2021Learning One Representation to Optimize All Rewardscited by 115
  34. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  35. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  36. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  37. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  38. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 10; 2025: 8; 2026: 7.

36 papers.

Sort:

By area:

  1. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  2. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
  3. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  4. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  5. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 18
  6. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  7. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  8. 2022Latent Variable Representation for Reinforcement Learningcited by 15
  9. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  10. 2023Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPscited by 14
  11. 2024Spectral Representation for Causal Estimation with Hidden Confounderscited by 12
  12. 2023Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learningcited by 10
24 more
  1. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  2. 2024A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learningcited by 7
  3. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  4. 2024Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpointcited by 6
  5. 2025Representation Learning via Non-Contrastive Mutual Informationcited by 5
  6. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  7. 2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 4
  8. 2023Towards a Better Understanding of Representation Dynamics under TD-learningcited by 3
  9. 2024Scalable spectral representations for multi-agent reinforcement learning in network MDPscited by 3
  10. 2025Which Features are Best for Successor Features?cited by 3
  11. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representationscited by 3
  12. 2025Spectral Representation-based Reinforcement Learningcited by 3
  13. 2023Representations and Exploration for Deep Reinforcement Learning using Singular Value Decompositioncited by 2
  14. 2023Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learningcited by 2
  15. 2024Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimationcited by 1
  16. 2025Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behaviorcited by 1
  17. 2026Impact of Connectivity on Laplacian Representations in Reinforcement Learningcited by 1
  18. 2026Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learningcited by 1
  19. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  20. 2026Learning to Adapt: Representation-Based Reinforcement Learning for Multi-Task Skill Transfercited by 1
  21. 2024Efficient Duple Perturbation Robustness in Low-rank MDPscited by 0
  22. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
  23. 2026Factorized Spectral Representations for Reinforcement Learningcited by 0
  24. 2026Learning Expressive and Compositional Motion Representation via Spectral Skillscited by 0