# Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery

arXiv: https://arxiv.org/abs/2308.13135 · HTML: https://mlsanity.com/paper/2308.13135/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 57 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [The Uncertainty Bellman Equation and Exploration](https://mlsanity.com/paper/1709.05380/) (2017)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning](https://mlsanity.com/paper/1901.04670/) (2019)
- [Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation](https://mlsanity.com/paper/2006.13189/) (2020)
- [Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model](https://mlsanity.com/paper/1503.02978/) (2015)

## Abstract (arXiv)

We propose a nonparametric additive model for estimating interpretable value functions in reinforcement learning, with an application in optimizing postoperative recovery through personalized, adaptive recommendations. While reinforcement learning has achieved significant success in various domains, recent methods often rely on black-box approaches such as neural networks, which hinder the examination of individual feature contributions to a decision-making policy. Our novel method offers a flexible technique for estimating action-value functions without explicit parametric assumptions, overcoming the limitations of the linearity assumption of classical algorithms. By incorporating local kernel regression and basis expansion, we obtain a sparse, additive representation of the action-value function, enabling local approximation and retrieval of nonlinear, independent contributions of select state features and the interactions between joint feature pairs. We validate our approach through a simulation study and apply it to spine disease recovery, uncovering recommendations aligned with clinical knowledge. This method bridges the gap between flexible machine learning techniques and the interpretability required in healthcare applications, paving the way for more personalized interventions.