mlsanityData to 7 Oct 2026

Latent Variable Representation for Reinforcement Learning

arXiv 2212.08765 ↗First posted 17 Dec 2022cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dynamics. On the other hand, it remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of RL. In this paper, we provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle in the face of uncertainty for exploration. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models. Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings. Empirically, we demonstrate superior performance over current state-of-the-art algorithms across various benchmarks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 53 of its 89 references that are in the map. First posted on arXiv 17 Dec 2022.

Builds on

53 of 89 references.

References · 53

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2019An Introduction to Variational Autoencoderscited by 997
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2018DeepMind Control Suitecited by 935
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
38 more
  1. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  5. 2018Is Q-learning Provably Efficient?cited by 485
  6. 2020Reinforcement Learning with Augmented Datacited by 452
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  8. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  10. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  11. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  12. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  13. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  14. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  15. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  16. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  17. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  18. 2021Reinforcement Learning with Prototypical Representationscited by 179
  19. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  20. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  21. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  22. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  23. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  24. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  25. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  26. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  27. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  28. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  29. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  30. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  31. 2016Deep Successor Reinforcement Learningcited by 72
  32. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  33. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  34. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  35. 2014Scalable Kernel Methods via Doubly Stochastic Gradientscited by 36
  36. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  37. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  38. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 2; 2026: 3.

14 papers.

Sort:

By area:

  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  3. 2023Representation Learning with Multi-Step Inverse Kinematics: An Efficient and Optimal Approach to Rich-Observation RLcited by 22
  4. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  5. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  6. 2024Spectral Representation for Causal Estimation with Hidden Confounderscited by 12
  7. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  8. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  9. 2024Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpointcited by 3
  10. 2025Spectral Representation-based Reinforcement Learningcited by 3
  11. 2024Efficient Duple Perturbation Robustness in Low-rank MDPscited by 0
  12. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
2 more
  1. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0
  2. 2026Learning Expressive and Compositional Motion Representation via Spectral Skillscited by 0