mlsanityData to 7 Oct 2026

SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

arXiv 2402.13147 ↗First posted 20 Feb 2024cs.LG cs.AIcited by 7

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 42 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

30 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching5 Mar
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  3. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Data27 Oct
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  5. 2020Primal Wasserstein Imitation Learning8 Jun
  6. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  7. 2019Imitation Learning from Imperfect Demonstration27 Jan

Other references · 23

  1. 2021Convex optimizationcited by 2,479
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  12. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
11 more
  1. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  3. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  4. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  5. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  6. 2017Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patientscited by 13
  7. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learningcited by 13
  8. 2019Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrationscited by 10
  9. 2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9
  10. 2023Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learningcited by 7
  11. 2023Learning from Pixels with Expert Observationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025Efficient Imitation under Misspecificationcited by 4
  2. 2025Hierarchical Vision Language Action Model Using Success and Failure Demonstrationscited by 2
  3. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrationscited by 0
  4. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  5. 2026Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitationcited by 0
  6. 2026Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewardscited by 0
  7. 2026Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  2. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning
  3. 2022Discriminator-Guided Model-Based Offline Imitation Learning
  4. 2024Offline Imitation Learning with Model-based Reverse Augmentation
  5. 2021Offline Inverse Reinforcement Learning
  6. 2022Model-based Offline Imitation Learning with Non-expert Data
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon