mlsanityData to 7 Oct 2026

Dual Generator Offline Reinforcement Learning

arXiv 2211.01471 ↗First posted 2 Nov 2022cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline RL, constraining the learned policy to remain close to the data is essential to prevent the policy from outputting out-of-distribution (OOD) actions with erroneously overestimated values. In principle, generative adversarial networks (GAN) can provide an elegant solution to do so, with the discriminator directly providing a probability that quantifies distributional shift. However, in practice, GAN-based offline RL methods have not performed as well as alternative approaches, perhaps because the generator is trained to both fool the discriminator and maximize return -- two objectives that can be at odds with each other. In this paper, we show that the issue of conflicting objectives can be resolved by training two generators: one that maximizes return, with the other capturing the ``remainder'' of the data distribution in the offline dataset, such that the mixture of the two is close to the behavior policy. We show that not only does having two generators enable an effective GAN-based offline RL method, but also approximates a support constraint, where the policy does not need to match the entire data distribution, but only the slice of the data that leads to high long term performance. We name our method DASCO, for Dual-Generator Adversarial Support Constrained Offline RL. On benchmark tasks that require learning from sub-optimal data, DASCO significantly outperforms prior methods that enforce distribution constraint.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 51 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

36 of 51 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  10. 2020Critic Regularized Regression26 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
3 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2021Convex optimizationcited by 2,479
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2016Least Squares Generative Adversarial Networkscited by 1,000
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
9 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  4. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  5. 2016Amortised MAP Inference for Image Super-resolutioncited by 68
  6. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  7. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  8. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximationcited by 11
  9. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Reward Gaming in Conditional Text Generationcited by 29

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Offline Reinforcement Learning with Inaccurate Simulators
  2. 2023Model-based Offline Policy Optimization with Adversarial Network
  3. 2024DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learning
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2021Offline Inverse Reinforcement Learning
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
  7. 2023Adversarial Model for Offline Reinforcement Learning
  8. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions