# Latent Utility Q-Learning for Preference-Adaptive Dynamic Treatment Regimes

arXiv: https://arxiv.org/abs/2307.12022 · HTML: https://mlsanity.com/paper/2307.12022/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Inverse Preference Learning: Preference-based RL without a Reward Function](https://mlsanity.com/paper/2305.15363/) (2023)
- [Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination](https://mlsanity.com/paper/2107.02237/) (2021)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [Statistical Inference for Online Decision Making via Stochastic Gradient Descent](https://mlsanity.com/paper/2010.07341/) (2020)

## Abstract (arXiv)

Optimizing individualized treatment sequences for patients who weigh multiple, competing outcomes differently poses a challenge for dynamic treatment regime (DTR) methods, which typically assume a single univariate outcome. We propose Latent Utility Q-Learning (LUQ-Learning), which estimates DTRs optimizing patient-specific preference-weighted combinations of multivariate outcomes Y∈ℝd across K decision points. A conditional mean factorization decouples preference estimation from outcome regression, enabling flexible, modular learning under imperfectly observed and heterogeneous preferences without requiring explicit outcome ranking by patients. We establish consistency of the estimated value function and derive unified ε-optimality guarantees that bound policy value loss in terms of posterior preference uncertainty, yielding interpretable criteria for data-driven policy selection. Simulations calibrated to Sequential Multiple Assignment Randomized Trials (SMARTs) demonstrate that LUQ-Learning outperforms Q-learning with naive outcome aggregation, last-reported satisfaction optimization, and existing preference-based methods.