Learning non-Markovian Decision-Making from State-only Sequences

arXiv 2306.15156 ↗First posted 27 Jun 2023cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Conventional imitation learning assumes access to the actions of demonstrators, but these motor signals are often non-observable in naturalistic settings. Additionally, sequential decision-making behaviors in these settings can deviate from the assumptions of a standard Markov Decision Process (MDP). To address these challenges, we explore deep generative modeling of state-only sequences with non-Markov Decision Process (nMDP), where the policy is an energy-based prior in the latent space of the state transition generator. We develop maximum likelihood estimation to achieve model-based imitation, which involves short-run MCMC sampling from the prior and importance sampling for the posterior. The learned model enables textitdecision-making as inference: model-free policy execution is equivalent to prior sampling, model-based planning is posterior sampling initialized from the policy. We demonstrate the efficacy of the proposed method in a prototypical path planning task with non-Markovian constraints and show that the learned model exhibits strong performances in challenging domains from the MuJoCo suite.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 75 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

36 of 75 references.

Closest ancestors · 6

Same direction, newest first

2021Implicit Behavioral Cloning1 Sep
2021Off-Policy Imitation Learning from Observations25 Feb
2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
2019Provably Efficient Imitation Learning from Observation Alone27 May
2018Generative Adversarial Imitation from Observation17 Jul
2018Behavioral Cloning from Observation4 May

Other references · 30

2020Language Models are Few-Shot Learnerscited by 32,419
2017Proximal Policy Optimization Algorithmscited by 15,215
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2016Generative Adversarial Imitation Learningcited by 1,597
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
2020MOReL : Model-Based Offline Reinforcement Learningcited by 590

18 more

2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
2020Reinforcement Learning with Augmented Datacited by 468
2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
2020Learning Latent Space Energy-Based Prior Modelcited by 104
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 73
2021On the Expressivity of Markov Rewardcited by 71
2018Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motioncited by 18
2022Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processescited by 10
2022Space is a latent sequence: Structured sequence learning as a unified theory of representation in the hippocampuscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

2024Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scalecited by 8
2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
2024Offline Hierarchical Reinforcement Learning via Inverse Optimizationcited by 4
2024AIGB: Generative Auto-bidding via Conditional Diffusion Modelingcited by 2
2025Generative Actor Criticcited by 2
2025Constructing Non-Markovian Decision Process via History Aggregatorcited by 1

Similar, not cited

Close in wording, no citation either way