mlsanityData to 7 Oct 2026

SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories

arXiv 2511.08136 ↗First posted 11 Nov 2025cs.LG cs.AI stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each timestep can be difficult. However, it is often feasible to collect trajectories reflecting undesirable or risky behavior, implicitly conveying the behavior the agent should avoid. We refer to these trajectories as non-preferred trajectories. Unlike standard IL, which aims to mimic demonstrations, our agent must also learn to avoid risky behavior using non-preferred trajectories. In this paper, we propose a novel approach, SafeMIL, to learn a parameterized cost that predicts if the state-action pair is risky via Multiple Instance Learning. The learned cost is then used to avoid non-preferred behaviors, resulting in a policy that prioritizes safety. We empirically demonstrate that our approach can learn a safer policy that satisfies cost constraints without degrading the reward performance, thereby outperforming several baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 31 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

15 of 31 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2017Deep Setscited by 1,394
  5. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  7. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 163
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
3 more
  1. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  3. 2020Combating False Negatives in Adversarial Imitation Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
  2. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  3. 2026TAIL-Safe: Task-Agnostic Safety Monitoring for Imitation Learning Policies
  4. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  5. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  6. 2023A Strong Baseline for Batch Imitation Learning
  7. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
  8. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories