mlsanityData to 14 Oct 2026

Non-Markovian policies occupancy measures

arXiv 2205.13950 ↗First posted 27 May 2022cs.LG cs.SY eess.SYcited by 0

Not on the map. Why?

Abstract (arXiv)

A central object of study in Reinforcement Learning (RL) is the Markovian policy, in which an agent's actions are chosen from a memoryless probability distribution, conditioned only on its current state. The family of Markovian policies is broad enough to be interesting, yet simple enough to be amenable to analysis. However, RL often involves more complex policies: ensembles of policies, policies over options, policies updated online, etc. Our main contribution is to prove that the occupancy measure of any non-Markovian policy, i.e., the distribution of transition samples collected with it, can be equivalently generated by a Markovian policy. This result allows theorems about the Markovian policy class to be directly extended to its non-Markovian counterpart, greatly simplifying proofs, in particular those involving replay buffers and datasets. We provide various examples of such applications to the field of Reinforcement Learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

29 of 58 references.

References · 29

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2018Behavioral Cloning from Observationcited by 492
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
17 more
  1. 2018Distributed Prioritized Experience Replaycited by 269
  2. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  5. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
  8. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  9. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  10. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  11. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 27
  12. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  13. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  14. 2019Safe Policy Improvement with an Estimated Baseline Policycited by 17
  15. 2019A Reduction from Reinforcement Learning to No-Regret Online Learningcited by 10
  16. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  17. 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5

Built on by

Not cited yet.