# SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

arXiv: https://arxiv.org/abs/2402.13147 · HTML: https://mlsanity.com/paper/2402.13147/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 30)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [AlgaeDICE: Policy Gradient from Arbitrary Experience](https://mlsanity.com/paper/1912.02074/) (2019)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [Inverse Preference Learning: Preference-based RL without a Reward Function](https://mlsanity.com/paper/2305.15363/) (2023)
- [panda-gym: Open-source goal-conditioned environments for robotic learning](https://mlsanity.com/paper/2106.13687/) (2021)
- [TRAIL: Near-Optimal Imitation Learning with Suboptimal Data](https://mlsanity.com/paper/2110.14770/) (2021)
- [Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching](https://mlsanity.com/paper/2303.02569/) (2023)
- [LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2202.13536/) (2022)
- [Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications](https://mlsanity.com/paper/2302.07549/) (2023)
- [Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients](https://mlsanity.com/paper/1712.00654/) (2017)
- [BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2110.00894/) (2021)
- [Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations](https://mlsanity.com/paper/1910.07294/) (2019)
- [Generalization through Diversity: Improving Unsupervised Environment Design](https://mlsanity.com/paper/2301.08025/) (2023)
- [Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2312.10385/) (2023)
- [Learning from Pixels with Expert Observations](https://mlsanity.com/paper/2306.13872/) (2023)

## Built on by (most cited first, 7 of 7)

- [Efficient Imitation under Misspecification](https://mlsanity.com/paper/2503.13162/) (2025)
- [Hierarchical Vision Language Action Model Using Success and Failure Demonstrations](https://mlsanity.com/paper/2512.03913/) (2025)
- [UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations](https://mlsanity.com/paper/2410.08307/) (2024)
- [Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations](https://mlsanity.com/paper/2505.21182/) (2025)
- [Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation](https://mlsanity.com/paper/2603.03778/) (2026)
- [Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards](https://mlsanity.com/paper/2604.00258/) (2026)
- [Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers](https://mlsanity.com/paper/2604.12509/) (2026)

## Abstract (arXiv)

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.