mlsanityData to 7 Oct 2026

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training

arXiv 2301.07421 ↗First posted 18 Jan 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a surrogate, steering policy optimization towards more valuable regions of the reward landscape thus learning an optimal policy. Our results show that DIRECT outperforms state-of-the-art algorithms in sparse- and shifting-reward environments being able to provide a surrogate reward to the policy and direct the optimization towards valuable areas.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

33 of 54 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2014Conditional Generative Adversarial Netscited by 2,152
  9. 2016Adversarial examples in the physical worldcited by 1,762
  10. 2016OpenAI Gymcited by 1,684
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
21 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2017Adversarial Attacks on Neural Network Policiescited by 314
  7. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  8. 2019Simple Black-box Adversarial Attackscited by 249
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  10. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  11. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  13. 2017Neural Episodic Controlcited by 148
  14. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  15. 2018Self-Imitation Learningcited by 135
  16. 2016Connecting Generative Adversarial Networks and Actor-Critic Methodscited by 49
  17. 2018Generative Adversarial Self-Imitation Learningcited by 25
  18. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  19. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 24
  20. 2020Stable Adversarial Learning under Distributional Shiftscited by 16
  21. 2019Independent Generative Adversarial Self-Imitation Learning in Cooperative Multiagent Systemscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2018Learning Self-Imitating Diverse Policies
  2. 2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble
  3. 2019Co-training for Policy Learning
  4. 2019Reward-Conditioned Policies
  5. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
  6. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling
  7. 2026DDO-RM: Distribution-Level Policy Improvement after Reward Learning
  8. 2019Learning Human Objectives by Evaluating Hypothetical Behavior