mlsanityData to 14 Oct 2026

Hyperparameter Selection for Imitation Learning

arXiv 2105.12034 ↗First posted 25 May 2021cs.LGcited by 17

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We address the issue of tuning hyperparameters (HPs) for imitation learning algorithms in the context of continuous-control, when the underlying reward function of the demonstrating expert cannot be observed at any time. The vast literature in imitation learning mostly considers this reward function to be available for HP selection, but this is not a realistic setting. Indeed, would this reward function be available, it could then directly be used for policy training and imitation would not be necessary. To tackle this mostly ignored problem, we propose a number of possible proxies to the external reward. We evaluate them in an extensive empirical study (more than 10'000 agents across 9 environments) and make practical recommendations for selecting HPs. Our results show that while imitation learning algorithms are sensitive to HP choices, it is often possible to select good enough HPs through a proxy to the reward function.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 57 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

28 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020Primal Wasserstein Imitation Learning8 Jun
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  3. 2019Imitation Learning as f-Divergence Minimization30 May
  4. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2016OpenAI Gymcited by 1,736
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
11 more
  1. 2017Population Based Training of Neural Networkscited by 367
  2. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  3. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  4. 2020First return, then explorecited by 266
  5. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  6. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  7. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  8. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  9. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 63
  10. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  11. 2020Predictive Information Accelerates Learning in RLcited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 2; 2024: 4; 2025: 1; 2026: 2.

17 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  2. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  3. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  4. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 24
  5. 2021Continuous Control with Action Quantization from Demonstrationscited by 22
  6. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  7. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  8. 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Taskscited by 10
  9. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  10. 2024A Comparison of Imitation Learning Algorithms for Bimanual Manipulationcited by 8
  11. 2024LoRD: Adapting Differentiable Driving Policies to Distribution Shiftscited by 4
  12. 2022Extraneousness-Aware Imitation Learningcited by 3
5 more
  1. 2025Model Predictive Adversarial Imitation Learning for Planning from Observationcited by 2
  2. 2026InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersonscited by 2
  3. 2023Quantum Imitation Learningcited by 1
  4. 2024Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstrationcited by 1
  5. 2026Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Imitation Learning by Reinforcement Learning
  2. 2023A Strong Baseline for Batch Imitation Learning
  3. 2022A Ranking Game for Imitation Learning
  4. 2019Reinforced Imitation in Heterogeneous Action Space
  5. 2021Imitation by Predicting Observations
  6. 2022Imitation Learning by Estimating Expertise of Demonstrators
  7. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
  8. 2023Selective Sampling and Imitation Learning via Online Regression