mlsanityData to 7 Oct 2026

Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

arXiv 2302.08560 ↗First posted 16 Feb 2023cs.LG cs.AI cs.ROcited by 48

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 98 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

62 of 98 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  2. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
  3. 2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping15 Sep
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2023Mastering Diverse Domains through World Modelscited by 1,193
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
38 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2021Implicit Behavioral Cloningcited by 442
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  7. 2018Maximum a Posteriori Policy Optimisationcited by 318
  8. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  10. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  11. 2022Online Decision Transformercited by 194
  12. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  13. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  14. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  16. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  17. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  18. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  19. 2022Jump-Start Reinforcement Learningcited by 102
  20. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  21. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  22. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  23. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  24. 2021Off-Policy Imitation Learning from Observationscited by 70
  25. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  26. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  27. 2020Quantifying Differences in Reward Functionscited by 62
  28. 2020Learning Off-Policy with Online Planningcited by 46
  29. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  30. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  31. 2020Logistic Q-Learningcited by 34
  32. 2022Proximal Point Imitation Learningcited by 25
  33. 2022A Ranking Game for Imitation Learningcited by 18
  34. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  35. 2014Linear Programming for Large-Scale Markov Decision Problemscited by 12
  36. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  37. 2022Change of measure through the Legendre transformcited by 10
  38. 2020Imitative Planning using Conditional Normalizing Flowcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 17; 2026: 14.

48 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  3. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  5. 2025Horizon Reduction Makes RL Scalablecited by 55
  6. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  7. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  8. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  9. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  10. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  11. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
  12. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
36 more
  1. 2024A Distributional Analogue to the Successor Representationcited by 15
  2. 2025Dual Goal Representationscited by 13
  3. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  5. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  6. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  7. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  8. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  9. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  10. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  11. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  12. 2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 4
  13. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  14. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  15. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  16. 2024TRAM: Test-Time Risk Adaptation with Mixture of Agentscited by 1
  17. 2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1
  18. 2025Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Mapscited by 1
  19. 2025Analysis of approximate linear programming solution to Markov decision problem with log barrier functioncited by 1
  20. 2026Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Modelscited by 1
  21. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  22. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  23. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 1
  24. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  25. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  26. 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learningcited by 0
  27. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  28. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rolloutscited by 0
  29. 2026General Flexible f-divergence for Challenging Offline RL Datasets with Low Stochasticity and Diverse Behavior Policiescited by 0
  30. 2026Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learningcited by 0
  31. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  32. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 0
  33. 2026Moment Matching Q-Learningcited by 0
  34. 2026Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learningcited by 0
  35. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  36. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  2. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  3. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
  7. 2020MOReL : Model-Based Offline Reinforcement Learning
  8. 2024Unsupervised-to-Online Reinforcement Learning