# Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling

arXiv: https://arxiv.org/abs/1907.05634 · HTML: https://mlsanity.com/paper/1907.05634/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 12 Jul 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 8 of 8)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Toward the Fundamental Limits of Imitation Learning](https://mlsanity.com/paper/2009.05990/) (2020)
- [Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally](https://mlsanity.com/paper/2102.12948/) (2021)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [DROMO: Distributionally Robust Offline Model-based Policy Optimization](https://mlsanity.com/paper/2109.07275/) (2021)
- [SR-Reward: Taking The Path More Traveled](https://mlsanity.com/paper/2501.02330/) (2025)
- [HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach](https://mlsanity.com/paper/2306.06329/) (2023)
- [Digital Twin Calibration with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.02205/) (2025)

## Abstract (arXiv)

Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively-extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform significantly prior works in sample efficiency.