mlsanityData to 7 Oct 2026

Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator

arXiv 2401.16772 ↗First posted 30 Jan 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning is often used in addition to reinforcement learning in environments where reward design is difficult or where the reward is sparse, but it is difficult to be able to imitate well in unknown states from a small amount of expert data and sampling data. Supervised learning methods such as Behavioral Cloning do not require sampling data, but usually suffer from distribution shift. The methods based on reinforcement learning, such as inverse reinforcement learning and Generative Adversarial imitation learning (GAIL), can learn from only a few expert data. However, they often need to interact with the environment. Soft Q imitation learning (SQIL) addressed the problems, and it was shown that it could learn efficiently by combining Behavioral Cloning and soft Q-learning with constant rewards. In order to make this algorithm more robust to distribution shift, we propose more efficient and robust algorithm by adding to this method a reward function based on adversarial inverse reinforcement learning that rewards the agent for performing actions in status similar to the demo. We call this algorithm Discriminator Soft Q Imitation Learning (DSQIL). We evaluated it on MuJoCo environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 23 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

13 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Imitation Learning with Sinkhorn Distances20 Aug

Other references · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021Convex optimizationcited by 2,479
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Deep Reinforcement Learning that Matterscited by 727
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2020TrajGAIL: Generating Urban Vehicle Trajectories using Generative Adversarial Imitation Learningcited by 46
  11. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28
  12. 2020Enhanced Behavioral Cloning Based self-driving Car Using Transfer Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Discriminator Soft Actor Critic without Extrinsic Rewards
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitation
  3. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  4. 2026Scalable Causal Imitation Learning
  5. 2023Coherent Soft Imitation Learning
  6. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
  8. 2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble