mlsanityData to 14 Oct 2026

An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning

arXiv 2504.13368 ↗First posted 17 Apr 2025cs.LG cs.AIcited by 4

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

We introduce Iterative Dual Reinforcement Learning (IDRL), a new method that takes an optimal discriminator-weighted imitation view of solving RL. Our method is motivated by a simple experiment in which we find training a discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weighted behavior cloning gives strong results on various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, we find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero-weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 67 of its 80 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

67 of 80 references.

References · 67

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2021Convex optimizationcited by 2,515
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
38 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  8. 2020Critic Regularized Regressioncited by 275
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  12. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  13. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  14. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  15. 2021Offline RL Without Off-Policy Evaluationcited by 151
  16. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  17. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  18. 2019Imitation Learning as f-Divergence Minimizationcited by 123
  19. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  20. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  21. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  22. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  23. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  24. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  25. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  26. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  27. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  28. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  29. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  30. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  31. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  32. 2021Off-Policy Imitation Learning from Observationscited by 70
  33. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  34. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  35. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  36. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  37. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  38. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  2. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  3. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  4. 2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
  2. 2023A Strong Baseline for Batch Imitation Learning
  3. 2025Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
  4. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning
  5. 2024Align Your Intents: Offline Imitation Learning via Optimal Transport
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  7. 2024Hybrid Inverse Reinforcement Learning
  8. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)