# Beyond Exponentially Discounted Sum: Automatic Learning of Return Function

arXiv: https://arxiv.org/abs/1905.11591 · HTML: https://mlsanity.com/paper/1905.11591/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 7 of 7)

- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [Meta-Gradients in Non-Stationary Environments](https://mlsanity.com/paper/2209.06159/) (2022)
- [One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/2111.00206/) (2021)
- [Adaptive Pairwise Weights for Temporal Credit Assignment](https://mlsanity.com/paper/2102.04999/) (2021)
- [Applications of Reinforcement Learning in Finance -- Trading with a Double Deep Q-Network](https://mlsanity.com/paper/2206.14267/) (2022)
- [Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning](https://mlsanity.com/paper/2112.08094/) (2021)
- [Acceleration in Policy Optimization](https://mlsanity.com/paper/2306.10587/) (2023)

## Abstract (arXiv)

In reinforcement learning, Return, which is the weighted accumulated future rewards, and Value, which is the expected return, serve as the objective that guides the learning of the policy. In classic RL, return is defined as the exponentially discounted sum of future rewards. One key insight is that there could be many feasible ways to define the form of the return function (and thus the value), from which the same optimal policy can be derived, yet these different forms might render dramatically different speeds of learning this policy. In this paper, we research how to modify the form of the return function to enhance the learning towards the optimal policy. We propose to use a general mathematical form for return function, and employ meta-learning to learn the optimal return function in an end-to-end manner. We test our methods on a specially designed maze environment and several Atari games, and our experimental results clearly indicate the advantages of automatically learning optimal return functions in reinforcement learning.