mlsanityData to 14 Oct 2026

Leveraging Fully Observable Policies for Learning under Partial Observability

arXiv 2211.01991 ↗First posted 3 Nov 2022cs.RO cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning in partially observable domains is challenging due to the lack of observable state information. Thankfully, learning offline in a simulator with such state information is often possible. In particular, we propose a method for partially observable reinforcement learning that uses a fully observable policy (which we call a state expert) during offline training to improve online performance. Based on Soft Actor-Critic (SAC), our agent balances performing actions similar to the state expert and getting high returns under partial observability. Our approach can leverage the fully-observable policy for exploration and parts of the domain that are fully observable while still being able to learn under partial observability. On six robotics domains, our method outperforms pure imitation, pure reinforcement learning, the sequential or parallel combination of both types, and a recent state-of-the-art method in the same setting. A successful policy transfer to a physical robot in a manipulation task from pixels shows our approach's practicality in learning interesting policies under partial observability.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

27 of 48 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2016OpenAI Gymcited by 1,736
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2016End to End Learning for Self-Driving Carscited by 1,009
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
15 more
  1. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  2. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
  3. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  4. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  5. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 54
  6. 2021Partially Observable Markov Decision Processes (POMDPs) and Roboticscited by 49
  7. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  8. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  9. 2020Discriminative Particle Filter Reinforcement Learning for Complex Partial Observationscited by 28
  10. 2021Unbiased Asymmetric Reinforcement Learning under Partial Observabilitycited by 25
  11. 2019Learning Belief Representations for Imitation Learning in POMDPscited by 24
  12. 2020Robust Asymmetric Learning in POMDPscited by 23
  13. 2020Belief-Grounded Networks for Accelerated Robot Learning under Partial Observabilitycited by 10
  14. 2022BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Frameworkcited by 9
  15. 2022Hierarchical Reinforcement Learning under Mixed Observabilitycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 4; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 25
  2. 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedbackcited by 24
  3. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 19
  4. 2024Equivariant Reinforcement Learning under Partial Observabilitycited by 18
  5. 2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 17
  6. 2023Informed POMDP: Leveraging Additional Information in Model-Based RLcited by 11
  7. 2023Bridging the Sim-to-Real Gap from the Information Bottleneck Perspectivecited by 10
  8. 2024Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wristcited by 9
  9. 2024Proximal Policy Distillationcited by 9
  10. 2024Student-Informed Teacher Trainingcited by 6
  11. 2024IGDrivSim: A Benchmark for the Imitation Gap in Autonomous Drivingcited by 3
  12. 2025Guided Policy Optimization under Partial Observabilitycited by 3
6 more
  1. 2025To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learningcited by 3
  2. 2024A Bayesian Solution To The Imitation Gapcited by 2
  3. 2025Dexterous Grasping with Real-World Robotic Reinforcement Learningcited by 2
  4. 2025Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policiescited by 2
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environmentscited by 1
  6. 2026Group-Invariant Unsupervised Skill Discovery: Symmetry-aware Skill Representations for Generalizable Behaviorcited by 0