Representations and Exploration for Deep Reinforcement Learning using Singular Value Decomposition
Not on the map. Why?
Abstract (arXiv)
Representation learning and exploration are among the key challenges for any deep reinforcement learning agent. In this work, we provide a singular value decomposition based method that can be used to obtain representations that preserve the underlying transition structure in the domain. Perhaps interestingly, we show that these representations also capture the relative frequency of state visitations, thereby providing an estimate for pseudo-counts for free. To scale this decomposition method to large-scale domains, we provide an algorithm that never requires building the transition matrix, can make use of deep networks, and also permits mini-batch training. Further, we draw inspiration from predictive state representations and extend our decomposition method to partially observable environments. With experiments on multi-task settings with partially observable domains, we show that the proposed method can not only learn useful representation on DM-Lab-30 environments (that have inputs involving language instructions, pixel images, and rewards, among others) but it can also be effective at hard exploration tasks in DM-Hard-8 environments.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 53 of its 81 references that are in the map. First posted on arXiv 1 May 2023.
Builds on
53 of 81 references.
References · 53
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2015Trust Region Policy Optimizationcited by 2,580
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
- 2018Exploration by Random Network Distillationcited by 993
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
- 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
- 2018Maximum a Posteriori Policy Optimisationcited by 328
38 more
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
- 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
- 2017Count-Based Exploration with Neural Density Modelscited by 257
- 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
- 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
- 2016Learning to Navigate in Complex Environmentscited by 198
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
- 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
- 2021Learning One Representation to Optimize All Rewardscited by 115
- 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
- 2018Count-Based Exploration with the Successor Representationcited by 112
- 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
- 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
- 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
- 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
- 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
- 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
- 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
- 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
- 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
- 2019Shaping Belief States with Generative Environment Models for RLcited by 66
- 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
- 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
- 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
- 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 52
- 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
- 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 44
- 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
- 2022PAC Reinforcement Learning for Predictive State Representationscited by 43
- 2020EigenGame: PCA as a Nash Equilibriumcited by 39
- 2016A PAC RL Algorithm for Episodic POMDPscited by 37
- 2022Spectral Decomposition Representation for Reinforcement Learningcited by 36
- 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
- 2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 27
- 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 23
- 2022Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiencycited by 16
- 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
- 2019World Discovery Modelscited by 10
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1.
2 papers.