mlsanityData to 7 Oct 2026

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

arXiv 1910.11956 ↗First posted 25 Oct 2019cs.LG cs.RO stat.MLcited by 380

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at https://relay-policy-learning.github.io/

Key facts. Cited by 380 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 25 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 7; 2021: 30; 2022: 52; 2023: 78; 2024: 95; 2025: 78; 2026: 40.

380 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  7. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  8. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  9. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  11. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  12. 2024Diffusion Policy Policy Optimizationcited by 255
38 more
  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  4. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  8. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  9. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  10. 2024Behavior Generation with Latent Actionscited by 159
  11. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  12. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  13. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  14. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  15. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  16. 2021Discovering and Achieving Goals via World Modelscited by 131
  17. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  18. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  19. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  20. 2023Parallel Sampling of Diffusion Modelscited by 112
  21. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  22. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  23. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  24. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  25. 2024Foundation Policies with Hilbert Representationscited by 89
  26. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  27. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  28. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  29. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  30. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  31. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  32. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  33. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  34. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  35. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  36. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  37. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  38. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72

50 most cited of 380 All 380 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  2. 2026SADP: Subgoal-Aware Diffusion Policy for Long-Horizon Manipulation Learned from Foundation Model Generated Demonstrations
  3. 2026Beyond the Majority: Long-tail Imitation Learning for Robotic Manipulation
  4. 2025textttSPIN: distilling textttSkill-RRT for long-horizon prehensile and non-prehensile manipulation ↗
  5. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulation
  6. 2026Training and Evaluating Diffusion Policies with Long Context Lengths
  7. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation
  8. 2025Learning Long-Context Diffusion Policies via Past-Token Prediction