Data to 7 Oct 2026

Offline Learning from Demonstrations and Unlabeled Experience

arXiv 2011.13885 ↗First posted 27 Nov 2020cs.LG cs.AI cs.ROcited by 72

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. However, BC does not effectively leverage what we will refer to as unlabeled experience: data of mixed and unknown quality without reward annotations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. Towards data-driven offline robot learning that can use this unlabeled experience, we introduce Offline Reinforced Imitation Learning (ORIL). ORIL first learns a reward function by contrasting observations from demonstrator and unlabeled trajectories, then annotates all data with the learned reward, and finally trains an agent via offline reinforcement learning. Across a diverse set of continuous control and simulated robotic manipulation tasks, we show that ORIL consistently outperforms comparable BC agents by effectively leveraging unlabeled experience.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 43 references that are in the map. First posted on arXiv 27 Nov 2020.

Builds on

28 of 43 references.

Closest ancestors · 6

Same direction, newest first

Other references · 22

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2016Generative Adversarial Imitation Learningcited by 1,576
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
2020Critic Regularized Regressioncited by 269
2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217

10 more

2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
2018Distributed Distributional Deterministic Policy Gradientscited by 195
2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
2017Learning human behaviors from motion capture by adversarial imitationcited by 75
2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
2019Positive-Unlabeled Reward Learningcited by 36
2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 10; 2023: 20; 2024: 17; 2025: 10; 2026: 6.

72 papers.

Sort:

By area:

2022A Generalist Agentcited by 802
2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
2023Optimal Transport for Offline Imitation Learningcited by 42
2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38

38 more

2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
2020Semi-supervised reward learning for offline reinforcement learningcited by 22
2023Coherent Soft Imitation Learningcited by 21
2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
2021Offline Preference-Based Apprenticeship Learningcited by 17
2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
2021Offline Inverse Reinforcement Learningcited by 14
2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 13
2023CEIL: Generalized Contextual Imitation Learningcited by 12
2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximationcited by 11
2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
2024DrS: Learning Reusable Dense Rewards for Multi-Stage Taskscited by 9
2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
2022Multi-Environment Pretraining Enables Transfer to Action Limited Datasetscited by 5
2023A Strong Baseline for Batch Imitation Learningcited by 5
2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
2024Robust Offline Imitation Learning from Diverse Auxiliary Datacited by 4
2023Programmatic Imitation Learning from Unlabeled and Noisy Demonstrationscited by 3
2023Contrastive Example-Based Controlcited by 3
2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way