# Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPs

arXiv: https://arxiv.org/abs/2505.12049 · HTML: https://mlsanity.com/paper/2505.12049/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 17 May 2025.

## Builds on (most cited first, 9 of 9)

- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?](https://mlsanity.com/paper/2502.15657/) (2025)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Settling the Reward Hypothesis](https://mlsanity.com/paper/2212.10420/) (2022)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Lexicographic Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2212.13769/) (2022)
- [Utility Theory for Sequential Decision Making](https://mlsanity.com/paper/2206.13637/) (2022)
- [On the Expressivity of Multidimensional Markov Reward](https://mlsanity.com/paper/2307.12184/) (2023)
- [Conditions on Preference Relations that Guarantee the Existence of Optimal Policies](https://mlsanity.com/paper/2311.01990/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Better Spur Should Start From Each Objective](https://mlsanity.com/paper/2609.08211/) (2026)
- [The syntax and semantics of goals](https://mlsanity.com/paper/2609.19448/) (2026)

## Abstract (arXiv)

Recent work has formalized the reward hypothesis through the lens of expected utility theory, by interpreting reward as utility. Hausner's foundational work showed that dropping the continuity axiom leads to a generalization of expected utility theory where utilities are lexicographically ordered vectors of arbitrary dimension. In this paper, we extend this result by identifying a simple and practical condition under which preferences cannot be represented by scalar rewards, necessitating a 2-dimensional reward function. We provide a full characterization of such reward functions, as well as the general d-dimensional case, in Markov Decision Processes (MDPs) under a memorylessness assumption on preferences. Furthermore, we show that optimal policies in this setting retain many desirable properties of their scalar-reward counterparts, while in the Constrained MDP (CMDP) setting -- another common multiobjective setting -- they do not.