mlsanityData to 14 Oct 2026

LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy

arXiv 2602.17312 ↗First posted 19 Feb 2026cs.LG cs.SY eess.SYcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 52 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

33 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning25 Dec
  2. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning12 Dec
  3. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning11 Dec
  4. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  5. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  6. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 27

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  5. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  6. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2021Reinforcement learning based recommender systems: A surveycited by 188
  12. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
15 more
  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  3. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  4. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  5. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  6. 2020A Review of Deep Reinforcement Learning for Smart Building Energy Managementcited by 38
  7. 2020Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPscited by 32
  8. 2019Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agentscited by 25
  9. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17
  10. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  11. 2024Thresholded Lexicographic Ordered Multiobjective Reinforcement Learningcited by 8
  12. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexitycited by 7
  13. 2023Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learningcited by 5
  14. 2024A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPscited by 5
  15. 2024GENESIS-RL: GEnerating Natural Edge-cases with Systematic Integration of Safety considerations and Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe-RULE: Safe Reinforcement UnLEarning
  2. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  3. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  4. 2026Safe Online Learning via Smooth Safety-Structured Policy Composition
  5. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  6. 2026Evaluation Metrics for Safe Reinforcement Learning
  7. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  8. 2024Evaluating Defences against Unsafe Feedback in RLHF