# Inference of Utilities and Time Preference in Sequential Decision-Making

arXiv: https://arxiv.org/abs/2405.15975 · HTML: https://mlsanity.com/paper/2405.15975/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 94 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 18 of 18)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces](https://mlsanity.com/paper/2310.02951/) (2023)
- [Towards Resolving Unidentifiability in Inverse Reinforcement Learning](https://mlsanity.com/paper/1601.06569/) (2016)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Robo-advising: Learning Investors' Risk Preferences via Portfolio Choices](https://mlsanity.com/paper/1911.02067/) (2019)
- [Robo-Advising: Enhancing Investment with Inverse Optimization and Deep Reinforcement Learning](https://mlsanity.com/paper/2105.09264/) (2021)
- [Personalized Robo-Advising: Enhancing Investment through Client Interaction](https://mlsanity.com/paper/1911.01391/) (2019)
- [Me, myself and I: a general theory of non-Markovian time-inconsistent stochastic control for sophisticated agents](https://mlsanity.com/paper/2002.12572/) (2020)
- [Towards Generalized Inverse Reinforcement Learning](https://mlsanity.com/paper/2402.07246/) (2024)

## Abstract (arXiv)

This paper introduces a novel stochastic control framework to enhance the capabilities of automated investment managers, or robo-advisors, by accurately inferring clients' investment preferences from past activities. Our approach leverages a continuous-time model that incorporates utility functions and a generic discounting scheme of a time-varying rate, tailored to each client's risk tolerance, valuation of daily consumption, and significant life goals. We address the resulting time inconsistency issue through state augmentation and the establishment of the dynamic programming principle and the verification theorem. Additionally, we provide sufficient conditions for the identifiability of client investment preferences. To complement our theoretical developments, we propose a learning algorithm based on maximum likelihood estimation within a discrete-time Markov Decision Process framework, augmented with entropy regularization. We prove that the log-likelihood function is locally concave, facilitating the fast convergence of our proposed algorithm. Practical effectiveness and efficiency are showcased through two numerical examples, including Merton's problem and an investment problem with unhedgeable risks. Our proposed framework not only advances financial technology by improving personalized investment advice but also contributes broadly to other fields such as healthcare, economics, and artificial intelligence, where understanding individual preferences is crucial.