mlsanityData to 7 Oct 2026

PLAS: Latent Action Space for Offline Reinforcement Learning

arXiv 2011.07213 ↗First posted 14 Nov 2020cs.RO cs.AI cs.LGcited by 133

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The goal of offline reinforcement learning is to learn a policy from a fixed dataset, without further interactions with the environment. This setting will be an increasingly more important paradigm for real-world applications of reinforcement learning such as robotics, in which data collection is slow and potentially dangerous. Existing off-policy algorithms have limited performance on static datasets due to extrapolation errors from out-of-distribution actions. This leads to the challenge of constraining the policy to select actions within the support of the dataset during training. We propose to simply learn the Policy in the Latent Action Space (PLAS) such that this requirement is naturally satisfied. We evaluate our method on continuous control benchmarks in simulation and a deformable object manipulation task with a physical robot. We demonstrate that our method provides competitive performance consistently across various continuous control tasks and different types of datasets, outperforming existing offline reinforcement learning methods with explicit constraints. Videos and code are available at https://sites.google.com/view/latent-policy.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 26 references that are in the map. First posted on arXiv 14 Nov 2020.

Builds on

18 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2016Tutorial on Variational Autoencoderscited by 404
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019Multi-Agent Reinforcement Learning with Multi-Step Generative Modelscited by 18
1 more
  1. 2017Prediction and Control with Temporal Segment Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 33; 2023: 27; 2024: 34; 2025: 14; 2026: 14.

133 papers.

Sort:

By area:

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  2. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  3. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  7. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  9. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  10. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  11. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  12. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
38 more
  1. 2023Anti-Exploration by Random Network Distillationcited by 49
  2. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  3. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  4. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  7. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 33
  8. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  9. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  10. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
  11. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  12. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  13. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 24
  14. 2022Learning Options via Compressioncited by 22
  15. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  16. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  17. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  18. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  19. 2022Visuotactile Affordances for Cloth Manipulation with Local Controlcited by 20
  20. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  21. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  22. 2023On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfercited by 20
  23. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  24. 2022Back to the Manifold: Recovering from Out-of-Distribution Statescited by 19
  25. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 18
  26. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  27. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  28. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  29. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  30. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  31. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  32. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  33. 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Datasetcited by 14
  34. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learningcited by 14
  35. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  36. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  37. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RLcited by 12
  38. 2023Generative Slate Recommendation with Reinforcement Learningcited by 12

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning from Images with Latent Space Models
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  3. 2020Contextual Latent-Movements Off-Policy Optimization for Robotic Manipulation Skills
  4. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  5. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets
  6. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling
  7. 2023Finetuning Offline World Models in the Real World
  8. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning