mlsanityData to 14 Oct 2026

AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL

arXiv 2606.31178 ↗First posted 30 Jun 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Optimizing nonlinear preferences in multi-objective reinforcement learning (MORL) is essential for capturing complex trade-offs like risk aversion or fairness. However, such non-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return (SER) and Expected Scalarized Return (ESR). While SER requires global-level optimization and ESR requires non-Markovian policies, leading to fragmented optimization strategies, we bridge this divide through the Aggregation-Expectation-Transformation (AET) framework. By unifying both criteria through a tripartite decomposition of scalarization, AET provides a principled foundation for general nonlinear MORL. Building on this framework, we propose AETDICE, a tractable offline RL algorithm for AET objectives. By utilizing DICE-style density-ratio estimation in an augmented state space, AETDICE enables sample-based optimization from static datasets. Our framework resolves long-standing barriers and captures respective trade-offs induced by AET framework, which existing methods fail to address.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 30 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

19 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning16 Jun
  2. 2024The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm12 Jun

Other references · 17

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  6. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  7. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  9. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  10. 2022Welfare and Fairness in Multi-objective Reinforcement Learningcited by 20
  11. 2024Policy-regularized Offline Multi-objective Reinforcement Learningcited by 10
  12. 2020A utility-based analysis of equilibria in multi-objective normal form gamescited by 9
5 more
  1. 2023Fairness in Preference-based Reinforcement Learningcited by 9
  2. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  3. 2023Multi-objective Reinforcement Learning with Nonlinear Preferences: Provable Approximation for Maximizing Expected Scalarized Returncited by 7
  4. 2021Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Makingcited by 6
  5. 2025FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
  2. 2024C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front
  3. 2025Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning
  4. 2025Limitations of Scalarisation in MORL: A Comparative Study in Discrete Environments
  5. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF
  6. 2026Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization
  7. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  8. 2025Benchmarking MOEAs for solving continuous multi-objective RL problems