mlsanityData to 7 Oct 2026

TRAIL: Near-Optimal Imitation Learning with Suboptimal Data

arXiv 2110.14770 ↗First posted 27 Oct 2021cs.LGcited by 39

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The aim in imitation learning is to learn effective policies by utilizing near-optimal expert demonstrations. However, high-quality demonstrations from human experts can be expensive to obtain in large numbers. On the other hand, it is often much easier to obtain large quantities of suboptimal or task-agnostic trajectories, which are not useful for direct imitation, but can nevertheless provide insight into the dynamical structure of the environment, showing what could be done in the environment even if not what should be done. We ask the question, is it possible to utilize such suboptimal offline datasets to facilitate provably improved downstream imitation learning? In this work, we answer this question affirmatively and present training objectives that use offline datasets to learn a factored transition model whose structure enables the extraction of a latent action space. Our theoretical analysis shows that the learned latent action space can boost the sample-efficiency of downstream imitation learning, effectively reducing the need for large near-optimal expert datasets through the use of auxiliary non-expert data. To learn the latent action space in practice, we propose TRAIL (Transition-Reparametrized Actions for Imitation Learning), an algorithm that learns an energy-based transition model contrastively, and uses the transition model to reparametrize the action space for sample-efficient imitation learning. We evaluate the practicality of our objective through experiments on a set of navigation and locomotion tasks. Our results verify the benefits suggested by our theory and show that TRAIL is able to improve baseline imitation learning by up to 4x in performance.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 62 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

39 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Visual Adversarial Imitation Learning using Variational Models16 Jul
  2. 2021Provable Representation Learning for Imitation with Contrastive Fourier Features26 May
  3. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  4. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimization24 Feb

Other references · 35

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2018DeepMind Control Suitecited by 935
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  9. 2016The Option-Critic Architecturecited by 433
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  12. 2019Learning Latent Plans from Playcited by 310
23 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  3. 2020Critic Regularized Regressioncited by 269
  4. 2016Variational Intrinsic Controlcited by 262
  5. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  6. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  7. 2021Reinforcement learning based recommender systems: A surveycited by 187
  8. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  9. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  10. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  11. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  12. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  13. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  14. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  15. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  16. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  17. 2017Multi-Level Discovery of Deep Optionscited by 50
  18. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  19. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39
  20. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  21. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  22. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  23. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 7; 2024: 7; 2025: 8; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  2. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  3. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  5. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  6. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  7. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  8. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  9. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  10. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  11. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  12. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 22
27 more
  1. 2022Learning Options via Compressioncited by 22
  2. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  3. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  4. 2022PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representationscited by 10
  5. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  6. 2023Learning to Discover Skills through Guidancecited by 8
  7. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
  8. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrationscited by 5
  9. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  10. 2022CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrationscited by 4
  11. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  12. 2025Efficient Imitation under Misspecificationcited by 4
  13. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representationscited by 3
  14. 2022Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrationscited by 2
  15. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 2
  16. 2023Boosting Reinforcement Learning and Planning with Demonstrations: A Surveycited by 1
  17. 2024ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Gamescited by 1
  18. 2024Learning Strategy Representation for Imitation Learning in Multi-Agent Gamescited by 1
  19. 2025NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrationscited by 1
  20. 2025Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behaviorcited by 1
  21. 2025Learning human-to-robot handovers through 3D scene reconstructioncited by 1
  22. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  23. 2024Boltzmann State-Dependent Rationalitycited by 0
  24. 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstratorscited by 0
  25. 2025Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learningcited by 0
  26. 2026A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDPcited by 0
  27. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  2. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning
  3. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  4. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  5. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  6. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  7. 2022Model-based Offline Imitation Learning with Non-expert Data
  8. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations