mlsanityData to 14 Oct 2026

An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning

arXiv 2409.09958 ↗First posted 16 Sep 2024cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

In recent years, significant progress has been made in multi-objective reinforcement learning (RL) research, which aims to balance multiple objectives by incorporating preferences for each objective. In most existing studies, specific preferences must be provided during deployment to indicate the desired policies explicitly. However, designing these preferences depends heavily on human prior knowledge, which is typically obtained through extensive observation of high-performing demonstrations with expected behaviors. In this work, we propose a simple yet effective offline adaptation framework for multi-objective RL problems without assuming handcrafted target preferences, but only given several demonstrations to implicitly indicate the preferences of expected policies. Additionally, we demonstrate that our framework can naturally be extended to meet constraints on safety-critical objectives by utilizing safe demonstrations, even when the safety thresholds are unknown. Empirical results on offline multi-objective and safe tasks demonstrate the capability of our framework to infer policies that align with real preferences while meeting the constraints implied by the provided demonstrations.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 43 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

31 of 43 references.

References · 31

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  7. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  8. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  9. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  10. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  11. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  12. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
19 more
  1. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  2. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
  3. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  4. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  5. 2016Multi-Objective Deep Reinforcement Learningcited by 83
  6. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  7. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  8. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  9. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 58
  10. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  11. 2020Inverse Constrained Reinforcement Learningcited by 46
  12. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  13. 2018Meta-Learning for Multi-objective Reinforcement Learningcited by 34
  14. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 28
  15. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 21
  16. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  17. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
  18. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 10
  19. 2024Policy-regularized Offline Multi-objective Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  2. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  3. 2026Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignmentcited by 0
  4. 2026Constrained Multi-Objective Reinforcement Learning with Max-Min Criterioncited by 0
  5. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning
  2. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  3. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
  4. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  5. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  6. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  7. 2025Online Optimization for Offline Safe Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory