# Imitating Cost-Constrained Behaviors in Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.17456 · HTML: https://mlsanity.com/paper/2403.17456/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Constrained Markov Decision Processes via Backward Value Functions](https://mlsanity.com/paper/2008.11811/) (2020)
- [Balancing Constraints and Rewards with Meta-Gradient D4PG](https://mlsanity.com/paper/2010.06324/) (2020)
- [Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2312.10385/) (2023)

## Built on by (most cited first, 1 of 1)

- [LLM-based Few-Shot Early Rumor Detection with Imitation Agent](https://mlsanity.com/paper/2512.18352/) (2025)

## Abstract (arXiv)

Complex planning and scheduling problems have long been solved using various optimization or heuristic approaches. In recent years, imitation learning that aims to learn from expert demonstrations has been proposed as a viable alternative to solving these problems. Generally speaking, imitation learning is designed to learn either the reward (or preference) model or directly the behavioral policy by observing the behavior of an expert. Existing work in imitation learning and inverse reinforcement learning has focused on imitation primarily in unconstrained settings (e.g., no limit on fuel consumed by the vehicle). However, in many real-world domains, the behavior of an expert is governed not only by reward (or preference) but also by constraints. For instance, decisions on self-driving delivery vehicles are dependent not only on the route preferences/rewards (depending on past demand data) but also on the fuel in the vehicle and the time available. In such problems, imitation learning is challenging as decisions are not only dictated by the reward model but are also dependent on a cost-constrained model. In this paper, we provide multiple methods that match expert distributions in the presence of trajectory cost constraints through (a) Lagrangian-based method; (b) Meta-gradients to find a good trade-off between expected return and minimizing constraint violation; and (c) Cost-violation-based alternating gradient. We empirically show that leading imitation learning approaches imitate cost-constrained behaviors poorly and our meta-gradient-based approach achieves the best performance.