mlsanityData to 14 Oct 2026

Approximate information state for approximate planning and reinforcement learning in partially observed systems

arXiv 2010.08843 ↗First posted 17 Oct 2020cs.LG cs.SY eess.SYcited by 70

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a theoretical framework for approximate planning and learning in partially observed systems. Our framework is based on the fundamental notion of information state. We provide two equivalent definitions of information state -- i) a function of history which is sufficient to compute the expected reward and predict its next value; ii) equivalently, a function of the history which can be recursively updated and is sufficient to compute the expected reward and predict the next observation. An information state always leads to a dynamic programming decomposition. Our key result is to show that if a function of the history (called approximate information state (AIS)) approximately satisfies the properties of the information state, then there is a corresponding approximate dynamic program. We show that the policy computed using this is approximately optimal with bounded loss of optimality. We show that several approximations in state, observation and action spaces in literature can be viewed as instances of AIS. In some of these cases, we obtain tighter bounds. A salient feature of AIS is that it can be learnt from data. We present AIS based multi-time scale policy gradient algorithms. and detailed numerical experiments with low, moderate and high dimensional environments.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 12; 2024: 15; 2025: 14; 2026: 19.

70 papers.

Sort:

By area:

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  2. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  3. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  4. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 18
  5. 2021Online Learning for Unknown Partially Observable MDPscited by 17
  6. 2023BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximationscited by 14
  7. 2022Finite-Time Analysis of Natural Actor-Critic for POMDPscited by 13
  8. 2023Informed POMDP: Leveraging Additional Information in Model-Based RLcited by 11
  9. 2022Reinforcement Learning in Non-Markovian Environmentscited by 10
  10. 2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
  11. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  12. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
38 more
  1. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  2. 2021Common Information based Approximate State Representations in Multi-Agent Reinforcement Learningcited by 8
  3. 2023Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver Behavior for Effective On-Ramp Mergingcited by 7
  4. 2023Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environmentscited by 7
  5. 2024Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approachcited by 7
  6. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  7. 2024Periodic agent-state based Q-learning for POMDPscited by 6
  8. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 6
  9. 2022On learning history based policies for controlling Markov decision processescited by 5
  10. 2023Approximate information state based convergence analysis of recurrent Q-learningcited by 5
  11. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  12. 2024A Framework for Effective AI Recommendations in Cyber-Physical-Human Systemscited by 4
  13. 2025Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatmentcited by 4
  14. 2022Towards Data-Driven Offline Simulations for Online Reinforcement Learningcited by 3
  15. 2022Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part Icited by 3
  16. 2023Approximate Information States for Worst-Case Control and Learning in Uncertain Systemscited by 3
  17. 2023Worst-Case Control and Learning Using Partial Observations Over an Infinite Time-Horizoncited by 3
  18. 2024Representation Learning For Efficient Deep Multi-Agent Reinforcement Learningcited by 3
  19. 2024Tractable Offline Learning of Regular Decision Processescited by 3
  20. 2025AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretabilitycited by 3
  21. 2024SplAgger: Split Aggregation for Meta-Reinforcement Learningcited by 2
  22. 2024Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPscited by 2
  23. 2025Learning Symbolic Persistent Macro-Actions for POMDP Solving Over Timecited by 2
  24. 2025Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Accesscited by 2
  25. 2026AURA: Action-Gated Memory for Robot Policies at Constant VRAMcited by 2
  26. 2024Model approximation in MDPs with unbounded per-step costcited by 1
  27. 2024Reinforcement Learning Design for Quickest Change Detectioncited by 1
  28. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  29. 2026Decoupling Return-to-Go for Efficient Decision Transformercited by 1
  30. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  31. 2026Model-Based Learning of Near-Optimal Finite-Window Policies in POMDPscited by 1
  32. 2026Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraintscited by 1
  33. 2026Exposure Bias as Epistemic Underidentification in Recursive Forecastingcited by 1
  34. 2022Dynamic Structure Estimation from Bandit Feedback using Nonvanishing Exponential Sumscited by 0
  35. 2022Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observabilitycited by 0
  36. 2023A Novel Point-based Algorithm for Multi-agent Control Using the Common Information Approachcited by 0
  37. 2023Deep Reinforcement Learning for Uplink Scheduling in NOMA-URLLC Networkscited by 0
  38. 2024Anticipating Oblivious Opponents in Stochastic Gamescited by 0

50 most cited of 70 All 70 (JSON)