mlsanityData to 7 Oct 2026

Learning Belief Representations for Imitation Learning in POMDPs

arXiv 1906.09510 ↗First posted 22 Jun 2019cs.LG stat.MLcited by 23

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider the problem of imitation learning from expert demonstrations in partially observable Markov decision processes (POMDPs). Belief representations, which characterize the distribution over the latent states in a POMDP, have been modeled using recurrent neural networks and probabilistic latent variable models, and shown to be effective for reinforcement learning in POMDPs. In this work, we investigate the belief representation learning problem for generative adversarial imitation learning in POMDPs. Instead of training the belief module and the policy separately as suggested in prior work, we learn the belief module jointly with the policy, using a task-aware imitation loss to ensure that the representation is more aligned with the policy's objective. To improve robustness of representation, we introduce several informative belief regularization techniques, including multi-step prediction of dynamics and action-sequences. Evaluated on various partially observable continuous-control locomotion tasks, our belief-module imitation learning approach (BMIL) substantially outperforms several baselines, including the original GAIL algorithm and the task-agnostic belief learning algorithm. Extensive ablation analysis indicates the effectiveness of task-aware belief learning and belief regularization.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 50 references that are in the map. First posted on arXiv 22 Jun 2019.

Builds on

29 of 50 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Variational Inference with Normalizing Flowscited by 2,161
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2017Convolutional Sequence to Sequence Learningcited by 550
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
17 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  2. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  3. 2016Learning to Navigate in Complex Environmentscited by 192
  4. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  5. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  6. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  7. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  8. 2016Sequential Neural Models with Stochastic Layerscited by 115
  9. 2017Z-Forcing: Training Stochastic Recurrent Networkscited by 51
  10. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  11. 2018Temporal Difference Variational Auto-Encodercited by 48
  12. 2014Online Planning Algorithms for POMDPscited by 42
  13. 2018Neural Predictive Belief Representationscited by 39
  14. 2016Learning to Act by Predicting the Futurecited by 36
  15. 2017Predictive-State Decoders: Encoding the Future into Recurrent Networkscited by 11
  16. 2018Playing Atari with Six Neuronscited by 11
  17. 2018Recurrent Predictive State Policy Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 4; 2022: 5; 2023: 4; 2024: 4; 2025: 3; 2026: 2.

23 papers.

Sort:

By area:

  1. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  2. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  3. 2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 17
  4. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  5. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  6. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  7. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delayscited by 10
  8. 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
  9. 2020Domain-Adversarial and Conditional State Space Model for Imitation Learningcited by 7
  10. 2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
  11. 2022Generalized Beliefs for Cooperative AIcited by 5
  12. 2021How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?cited by 4
11 more
  1. 2021Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learningcited by 4
  2. 2023Approximate Model-Based Shielding for Safe Reinforcement Learningcited by 4
  3. 2022Latent Policies for Adversarial Imitation Learningcited by 3
  4. 2022Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrationscited by 2
  5. 2022Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learningcited by 1
  6. 2024Learning for Long-Horizon Planning via Neuro-Symbolic Abductive Imitationcited by 1
  7. 2025Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videoscited by 1
  8. 2025BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulationcited by 0
  9. 2025Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidancecited by 0
  10. 2026BEACON: Belief-Enabled Adaptive CONtrol for Imitation Learning under Uncertaintycited by 0
  11. 2026Minimal Recurrent Behavioral Memory for Imitation under Partial Observabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  2. 2023Regularizing Adversarial Imitation Learning Using Causal Invariance
  3. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimization
  4. 2020State-only Imitation with Transition Dynamics Mismatch
  5. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations
  6. 2026Difference-Aware Retrieval Policies for Imitation Learning
  7. 2022An Empirical Investigation of Representation Learning for Imitation
  8. 2023Sample-efficient Adversarial Imitation Learning