# Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction

arXiv: https://arxiv.org/abs/1905.11100 · HTML: https://mlsanity.com/paper/1905.11100/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning with a Disentangled Universal Value Function for Item Recommendation](https://mlsanity.com/paper/2104.02981/) (2021)

## Abstract (arXiv)

Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from the perspective of future prediction, through decomposing the value function into a reward-independent future dynamics part and a policy-independent trajectory return part. We then derive a practical deep RL algorithm from the above decomposition, consisting of a convolutional trajectory representation model, a conditional variational dynamics model to predict the expected representation of future trajectory and a convex trajectory return model that maps a trajectory representation to its return. Our algorithm is evaluated in MuJoCo continuous control tasks and shows superior results under both common settings and delayed reward settings.