# Sufficient Markov Decision Processes with Alternating Deep Neural Networks

arXiv: https://arxiv.org/abs/1704.07531 · HTML: https://mlsanity.com/paper/1704.07531/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2017.

## Built on by (most cited first, 3 of 3)

- [Sequential Knockoffs for Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2303.14281/) (2023)
- [Off-policy Evaluation with Deeply-abstracted States](https://mlsanity.com/paper/2406.19531/) (2024)
- [Deep Spectral Q-learning with Application to Mobile Health](https://mlsanity.com/paper/2301.00927/) (2023)

## Abstract (arXiv)

Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying decision process that is both Markov and low-dimensional is non-trivial. We propose a method for constructing a low-dimensional representation of the original decision process for which: 1. the MDP model holds; 2. a decision strategy that maximizes mean utility when applied to the low-dimensional representation also maximizes mean utility when applied to the original process. We use a deep neural network to define a class of potential process representations and estimate the process of lowest dimension within this class. The method is illustrated using data from a mobile study on heavy drinking and smoking among college students.