# Reinforcement Learning with Non-Cumulative Objective

arXiv: https://arxiv.org/abs/2307.04957 · HTML: https://mlsanity.com/paper/2307.04957/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 43 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Hedging](https://mlsanity.com/paper/1802.03042/) (2018)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes](https://mlsanity.com/paper/1910.07072/) (2019)
- [Maximum Reward Formulation In Reinforcement Learning](https://mlsanity.com/paper/2010.03744/) (2020)
- [Distributed Pricing-Based User Association for Downlink Heterogeneous Cellular Networks](https://mlsanity.com/paper/1407.4694/) (2014)
- [Scalable Deep Reinforcement Learning for Routing and Spectrum Access in Physical Layer](https://mlsanity.com/paper/2012.11783/) (2020)
- [Max-Min Fairness in IRS-Aided Multi-Cell MISO Systems with Joint Transmit and Reflective Beamforming](https://mlsanity.com/paper/2003.00906/) (2020)

## Built on by (most cited first, 4 of 4)

- [To the Max: Reinventing Reward in Reinforcement Learning](https://mlsanity.com/paper/2402.01361/) (2024)
- [Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations](https://mlsanity.com/paper/2506.16016/) (2025)
- [Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning](https://mlsanity.com/paper/2405.13609/) (2024)
- [Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines](https://mlsanity.com/paper/2510.27329/) (2025)

## Abstract (arXiv)

In reinforcement learning, the objective is almost always defined as a cumulative function over the rewards along the process. However, there are many optimal control and reinforcement learning problems in various application fields, especially in communications and networking, where the objectives are not naturally expressed as summations of the rewards. In this paper, we recognize the prevalence of non-cumulative objectives in various problems, and propose a modification to existing algorithms for optimizing such objectives. Specifically, we dive into the fundamental building block for many optimal control and reinforcement learning algorithms: the Bellman optimality equation. To optimize a non-cumulative objective, we replace the original summation operation in the Bellman update rule with a generalized operation corresponding to the objective. Furthermore, we provide sufficient conditions on the form of the generalized operation as well as assumptions on the Markov decision process under which the globally optimal convergence of the generalized Bellman updates can be guaranteed. We demonstrate the idea experimentally with the bottleneck objective, i.e., the objectives determined by the minimum reward along the process, on classical optimal control and reinforcement learning tasks, as well as on two network routing problems on maximizing the flow rates.