# Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2312.10385 · HTML: https://mlsanity.com/paper/2312.10385/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2005.12964/) (2020)
- [Constrained Markov Decision Processes via Backward Value Functions](https://mlsanity.com/paper/2008.11811/) (2020)
- [Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning](https://mlsanity.com/paper/2009.13891/) (2020)
- [Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation](https://mlsanity.com/paper/2103.10693/) (2021)
- [Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients](https://mlsanity.com/paper/1712.00654/) (2017)
- [Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations](https://mlsanity.com/paper/1910.07294/) (2019)
- [Learning from Pixels with Expert Observations](https://mlsanity.com/paper/2306.13872/) (2023)

## Built on by (most cited first, 7 of 7)

- [Offline Safe Reinforcement Learning Using Trajectory Classification](https://mlsanity.com/paper/2412.15429/) (2024)
- [SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning](https://mlsanity.com/paper/2402.13147/) (2024)
- [Safety through feedback in Constrained RL](https://mlsanity.com/paper/2406.19626/) (2024)
- [Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs](https://mlsanity.com/paper/2412.06843/) (2024)
- [Imitating Cost-Constrained Behaviors in Reinforcement Learning](https://mlsanity.com/paper/2403.17456/) (2024)
- [UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations](https://mlsanity.com/paper/2410.08307/) (2024)
- [Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations](https://mlsanity.com/paper/2505.21182/) (2025)

## Abstract (arXiv)

A popular framework for enforcing safe actions in Reinforcement Learning (RL) is Constrained RL, where trajectory based constraints on expected cost (or other cost measures) are employed to enforce safety and more importantly these constraints are enforced while maximizing expected reward. Most recent approaches for solving Constrained RL convert the trajectory based cost constraint into a surrogate problem that can be solved using minor modifications to RL methods. A key drawback with such approaches is an over or underestimation of the cost constraint at each state. Therefore, we provide an approach that does not modify the trajectory based cost constraint and instead imitates ``good'' trajectories and avoids ``bad'' trajectories generated from incrementally improving policies. We employ an oracle that utilizes a reward threshold (which is varied with learning) and the overall cost constraint to label trajectories as ``good'' or ``bad''. A key advantage of our approach is that we are able to work from any starting policy or set of trajectories and improve on it. In an exhaustive set of experiments, we demonstrate that our approach is able to outperform top benchmark approaches for solving Constrained RL problems, with respect to expected cost, CVaR cost, or even unknown cost constraints.