mlsanityData to 7 Oct 2026

Rethinking ValueDice: Does It Really Improve Performance?

arXiv 2202.02468 ↗First posted 5 Feb 2022cs.LGcited by 15

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Since the introduction of GAIL, adversarial imitation learning (AIL) methods attract lots of research interests. Among these methods, ValueDice has achieved significant improvements: it beats the classical approach Behavioral Cloning (BC) under the offline setting, and it requires fewer interactions than GAIL under the online setting. Are these improvements benefited from more advanced algorithm designs? We answer this question by the following conclusions. First, we show that ValueDice could reduce to BC under the offline setting. Second, we verify that overfitting exists and regularization matters in the low-data regime. Specifically, we demonstrate that with weight decay, BC also nearly matches the expert performance as ValueDice does. The first two claims explain the superior offline performance of ValueDice. Third, we establish that ValueDice does not work when the expert trajectory is subsampled. Instead, the mentioned success of ValueDice holds when the expert trajectory is complete, in which ValueDice is closely related to BC that performs well as mentioned. Finally, we discuss the implications of our research for imitation learning studies beyond ValueDice.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 49 references that are in the map. First posted on arXiv 5 Feb 2022.

Builds on

26 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021On Generalization of Adversarial Imitation Learning and Beyond19 Jun
  2. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching6 Jun
  3. 2021What Matters for Adversarial Imitation Learning?1 Jun
  4. 2021Hyperparameter Selection for Imitation Learning25 May
  5. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  6. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  7. 2020Error Bounds of Imitating Policies and Environments22 Oct
  8. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  9. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  10. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  11. 2019Imitation Learning as f-Divergence Minimization30 May
  12. 2018An Algorithmic Perspective on Imitation Learning16 Nov
1 more
  1. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  9. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  10. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  11. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  12. 2019Target-Based Temporal Difference Learningcited by 20
1 more
  1. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 8; 2024: 3; 2025: 1.

15 papers.

Sort:

By area:

  1. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  2. 2023Coherent Soft Imitation Learningcited by 21
  3. 2022Planning for Sample Efficient Imitation Learningcited by 20
  4. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  5. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  6. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13
  7. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  8. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
  9. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  10. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  11. 2024OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequencescited by 3
  12. 2023Theoretical Analysis of Offline Imitation With Supplementary Datasetcited by 2
3 more
  1. 2025Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrationscited by 2
  2. 2023Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularizationcited by 1
  3. 2024Offline Imitation Learning by Controlling the Effective Planning Horizoncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
  2. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
  3. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism
  4. 2022Model-based Offline Imitation Learning with Non-expert Data
  5. 2021A Pragmatic Look at Deep Imitation Learning
  6. 2019On Value Discrepancy of Imitation Learning
  7. 2022ABC: Adversarial Behavioral Cloning for Offline Mode-Seeking Imitation Learning
  8. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms