mlsanityData to 14 Oct 2026

Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPs

arXiv 2505.12049 ↗First posted 17 May 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent work has formalized the reward hypothesis through the lens of expected utility theory, by interpreting reward as utility. Hausner's foundational work showed that dropping the continuity axiom leads to a generalization of expected utility theory where utilities are lexicographically ordered vectors of arbitrary dimension. In this paper, we extend this result by identifying a simple and practical condition under which preferences cannot be represented by scalar rewards, necessitating a 2-dimensional reward function. We provide a full characterization of such reward functions, as well as the general d-dimensional case, in Markov Decision Processes (MDPs) under a memorylessness assumption on preferences. Furthermore, we show that optimal policies in this setting retain many desirable properties of their scalar-reward counterparts, while in the Constrained MDP (CMDP) setting -- another common multiobjective setting -- they do not.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 9 of its 30 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

9 of 30 references.

References · 9

  1. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  2. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 75
  3. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  4. 2022Settling the Reward Hypothesiscited by 33
  5. 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
  6. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17
  7. 2022Utility Theory for Sequential Decision Makingcited by 8
  8. 2023On the Expressivity of Multidimensional Markov Rewardcited by 4
  9. 2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Better Spur Should Start From Each Objectivecited by 1
  2. 2026The syntax and semantics of goalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs
  2. 2026Generalised Bellman recurrence and three dualities in sequential decision-making
  3. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
  4. 2024Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
  5. 2026Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
  6. 2025Regret Analysis of Average-Reward Unichain MDPs via an Actor-Critic Approach
  7. 2024On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
  8. 2024Learning Utilities from Demonstrations in Markov Decision Processes