# Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.13609 · HTML: https://mlsanity.com/paper/2405.13609/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 22 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning](https://mlsanity.com/paper/2112.12288/) (2021)
- [Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraints](https://mlsanity.com/paper/1410.6445/) (2014)
- [Generating Realistic Stock Market Order Streams](https://mlsanity.com/paper/2006.04212/) (2020)
- [Graph-theoretic Simplification of Quantum Circuits with the ZX-calculus](https://mlsanity.com/paper/1902.03178/) (2019)
- [Maximum Reward Formulation In Reinforcement Learning](https://mlsanity.com/paper/2010.03744/) (2020)
- [To the Max: Reinventing Reward in Reinforcement Learning](https://mlsanity.com/paper/2402.01361/) (2024)
- [Optimizing ZX-Diagrams with Deep Reinforcement Learning](https://mlsanity.com/paper/2311.18588/) (2023)
- [Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization](https://mlsanity.com/paper/2206.06827/) (2022)
- [Reinforcement Learning with Non-Cumulative Objective](https://mlsanity.com/paper/2307.04957/) (2023)

## Abstract (arXiv)

Markov decision processes (MDPs) are used to model a wide variety of applications ranging from game playing over robotics to finance. Their optimal policy typically maximizes the expected sum of rewards given at each step of the decision process. However, a large class of problems does not fit straightforwardly into this framework: Non-cumulative Markov decision processes (NCMDPs), where instead of the expected sum of rewards, the expected value of an arbitrary function of the rewards is maximized. Example functions include the maximum of the rewards or their mean divided by their standard deviation. In this work, we introduce a general mapping of NCMDPs to standard MDPs. This allows all techniques developed to find optimal policies for MDPs, such as reinforcement learning or dynamic programming, to be directly applied to the larger class of NCMDPs. Focusing on reinforcement learning, we show applications in a diverse set of tasks, including classical control, portfolio optimization in finance, and discrete optimization problems. Given our approach, we can improve both final performance and training time compared to relying on standard MDPs.