# General non-linear Bellman equations

arXiv: https://arxiv.org/abs/1907.03687 · HTML: https://mlsanity.com/paper/1907.03687/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 8 of 8)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Return-based Scaling: Yet Another Normalisation Trick for Deep RL](https://mlsanity.com/paper/2105.05347/) (2021)
- [Reinforcement Learning with Non-Exponential Discounting](https://mlsanity.com/paper/2209.13413/) (2022)
- [Taylor Expansion of Discount Factors](https://mlsanity.com/paper/2106.06170/) (2021)
- [Orchestrated Value Mapping for Reinforcement Learning](https://mlsanity.com/paper/2203.07171/) (2022)
- [Conjugated Discrete Distributions for Distributional Reinforcement Learning](https://mlsanity.com/paper/2112.07424/) (2021)
- [An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models](https://mlsanity.com/paper/2404.15518/) (2024)

## Abstract (arXiv)

We consider a general class of non-linear Bellman equations. These open up a design space of algorithms that have interesting properties, which has two potential advantages. First, we can perhaps better model natural phenomena. For instance, hyperbolic discounting has been proposed as a mathematical model that matches human and animal data well, and can therefore be used to explain preference orderings. We present a different mathematical model that matches the same data, but that makes very different predictions under other circumstances. Second, the larger design space can perhaps lead to algorithms that perform better, similar to how discount factors are often used in practice even when the true objective is undiscounted. We show that many of the resulting Bellman operators still converge to a fixed point, and therefore that the resulting algorithms are reasonable and inherit many beneficial properties of their linear counterparts.