mlsanityData to 7 Oct 2026

Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning

arXiv 2311.12244 ↗First posted 20 Nov 2023cs.LG cs.AI stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

In most real-world reinforcement learning applications, state information is only partially observable, which breaks the Markov decision process assumption and leads to inferior performance for algorithms that conflate observations with state. Partially Observable Markov Decision Processes (POMDPs), on the other hand, provide a general framework that allows for partial observability to be accounted for in learning, exploration and planning, but presents significant computational and statistical challenges. To address these difficulties, we develop a representation-based perspective that leads to a coherent framework and tractable algorithmic approach for practical reinforcement learning from partial observations. We provide a theoretical analysis for justifying the statistical efficiency of the proposed algorithm, and also empirically demonstrate the proposed algorithm can surpass state-of-the-art performance with partial observations across various benchmarks, advancing reliable reinforcement learning towards more practical applications.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 61 of its 93 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

61 of 93 references.

References · 61

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2018DeepMind Control Suitecited by 935
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
38 more
  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  2. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  5. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  6. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  7. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  8. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  9. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  10. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  11. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  12. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  13. 2022Masked World Models for Visual Controlcited by 169
  14. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  15. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  16. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  17. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  18. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  19. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  20. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  21. 2015Memory-based control with recurrent neural networkscited by 77
  22. 2016Deep Successor Reinforcement Learningcited by 72
  23. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  24. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  25. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  26. 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 51
  27. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  28. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  29. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 43
  30. 2014Online Planning Algorithms for POMDPscited by 42
  31. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  32. 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
  33. 2018Neural Predictive Belief Representationscited by 39
  34. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  35. 2016A PAC RL Algorithm for Episodic POMDPscited by 37
  36. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
  37. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  38. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 6.

10 papers.

  1. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  2. 2024Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learningcited by 9
  3. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  4. 2025Spectral Representation-based Reinforcement Learningcited by 3
  5. 2026Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learningcited by 1
  6. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  7. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
  8. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  9. 2026A Decentralized Partially Observable Team Decision Methodology with Delayed Information Sharingcited by 0
  10. 2026Learning Expressive and Compositional Motion Representation via Spectral Skillscited by 0