mlsanityData to 7 Oct 2026

Latent Plans for Task-Agnostic Offline Reinforcement Learning

arXiv 2209.08959 ↗First posted 19 Sep 2022cs.RO cs.AI cs.CVcited by 81

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Everyday tasks of long-horizon and comprising a sequence of multiple implicit subtasks still impose a major challenge in offline robot control. While a number of prior methods aimed to address this setting with variants of imitation and offline reinforcement learning, the learned behavior is typically narrow and often struggles to reach configurable long-horizon goals. As both paradigms have complementary strengths and weaknesses, we propose a novel hierarchical approach that combines the strengths of both methods to learn task-agnostic long-horizon policies from high-dimensional camera observations. Concretely, we combine a low-level policy that learns latent skills via imitation learning and a high-level policy learned from offline reinforcement learning for skill-chaining the latent behavior priors. Experiments in various simulated and real robot control tasks show that our formulation enables producing previously unseen combinations of skills to reach temporally extended goals by "stitching" together latent skills through goal chaining with an order-of-magnitude improvement in performance upon state-of-the-art baselines. We even learn one multi-task visuomotor policy for 25 distinct manipulation tasks in the real world which outperforms both imitation learning and offline reinforcement learning techniques.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 55 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

49 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct

Other references · 47

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2020Mastering Atari with Discrete World Modelscited by 990
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
35 more
  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2016The Option-Critic Architecturecited by 433
  4. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  5. 2019Learning Latent Plans from Playcited by 310
  6. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  7. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  8. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  10. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  11. 2019Planning with Goal-Conditioned Policiescited by 144
  12. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  13. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  14. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  15. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  16. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  17. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 75
  18. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  19. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  20. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  21. 2022Skill-based Model-based Reinforcement Learningcited by 50
  22. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  23. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39
  24. 2018Time-Agnostic Prediction: Predicting Predictable Video Framescited by 37
  25. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  26. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
  27. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  28. 2020Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planningcited by 26
  29. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  30. 2019Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulationcited by 22
  31. 2019Keyframing the Future: Keyframe Discovery for Visual Prediction and Planningcited by 20
  32. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  33. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  34. 2022Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learningcited by 15
  35. 2021Hierarchical Policies for Cluttered-Scene Grasping with Latent Planscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 19; 2024: 29; 2025: 18; 2026: 13.

81 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  4. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  5. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  8. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  9. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  10. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  11. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  12. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
38 more
  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  2. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  3. 2025Unified Vision-Language-Action Modelcited by 128
  4. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  5. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  6. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  7. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  8. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  9. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  11. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  12. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  13. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  14. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  15. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  16. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  17. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  18. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  19. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  20. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  21. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  22. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  23. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  24. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  25. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  26. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  27. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  28. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  29. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Datacited by 20
  30. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  31. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  32. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  33. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 16
  34. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  35. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  36. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
  37. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  38. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation
  2. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
  3. 2026Latent Policy Steering through One-Step Flow Policies
  4. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks
  5. 2023Offline Skill Generalization via Task and Motion Planning
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  7. 2026Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning
  8. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching