mlsanityData to 14 Oct 2026

D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning

arXiv 2210.14428 ↗First posted 26 Oct 2022cs.LG cs.AI cs.HCcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

While combining imitation learning (IL) and reinforcement learning (RL) is a promising way to address poor sample efficiency in autonomous behavior acquisition, methods that do so typically assume that the requisite behavior demonstrations are provided by an expert that behaves optimally with respect to a task reward. If, however, suboptimal demonstrations are provided, a fundamental challenge appears in that the demonstration-matching objective of IL conflicts with the return-maximization objective of RL. This paper introduces D-Shape, a new method for combining IL and RL that uses ideas from reward shaping and goal-conditioned RL to resolve the above conflict. D-Shape allows learning from suboptimal demonstrations while retaining the ability to find the optimal policy with respect to the task reward. We experimentally validate D-Shape in sparse-reward gridworld domains, showing that it both improves over RL in terms of sample efficiency and converges consistently to the optimal policy in the presence of suboptimal demonstrations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 43 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

24 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  2. 2019Goal-conditioned Imitation Learning13 Jun
  3. 2019Recent Advances in Imitation Learning from Observation30 May
  4. 2019Reinforced Imitation in Heterogeneous Action Space6 Apr
  5. 2019Hybrid Reinforcement Learning with Expert State Sequences11 Mar
  6. 2019Imitation Learning from Imperfect Demonstration27 Jan
  7. 2018Generative Adversarial Imitation from Observation17 Jul
  8. 2018Behavioral Cloning from Observation4 May

Other references · 16

  1. 2016Generative Adversarial Imitation Learningcited by 1,597
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  5. 2020First return, then explorecited by 266
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  7. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  8. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  9. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  10. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  11. 2017Time Limits in Reinforcement Learningcited by 79
  12. 2019Learning from Trajectories via Subgoal Discoverycited by 28
4 more
  1. 2019RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstrationcited by 19
  2. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14
  3. 2018One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RLcited by 11
  4. 2021Plan-Based Relaxed Reward Shaping for Goal-Directed Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2026Automating Potential-based Reward Shaping with Vision Language Model Guidancecited by 2
  2. 2024Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  2. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks
  3. 2022Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
  4. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  5. 2026Learning from Demonstrations via Capability-Aware Goal Sampling
  6. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics
  7. 2022Self-Imitation Learning from Demonstrations
  8. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation