# Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making

arXiv: https://arxiv.org/abs/2002.01751 · HTML: https://mlsanity.com/paper/2002.01751/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2020.

## Built on by (most cited first, 30 of 30)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Deeply-Debiased Off-Policy Interval Estimation](https://mlsanity.com/paper/2105.04646/) (2021)
- [Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://mlsanity.com/paper/2202.10589/) (2022)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Projected State-action Balancing Weights for Offline Reinforcement Learning](https://mlsanity.com/paper/2109.04640/) (2021)
- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [Testing Stationarity and Change Point Detection in Reinforcement Learning](https://mlsanity.com/paper/2203.01707/) (2022)
- [Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data](https://mlsanity.com/paper/2210.08642/) (2022)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [Testing for the Markov Property in Time Series via Deep Conditional Generative Learning](https://mlsanity.com/paper/2305.19244/) (2023)
- [Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimization](https://mlsanity.com/paper/2301.02220/) (2023)
- [Sequential Knockoffs for Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2303.14281/) (2023)
- [Off-policy Evaluation in Doubly Inhomogeneous Environments](https://mlsanity.com/paper/2306.08719/) (2023)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning](https://mlsanity.com/paper/2412.05783/) (2024)
- [Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing](https://mlsanity.com/paper/2501.06366/) (2025)
- [A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Markets](https://mlsanity.com/paper/2202.10574/) (2022)
- [Doubly Inhomogeneous Reinforcement Learning](https://mlsanity.com/paper/2211.03983/) (2022)
- [Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences](https://mlsanity.com/paper/2407.17910/) (2024)
- [Statistical Inference in Reinforcement Learning: A Selective Survey](https://mlsanity.com/paper/2502.16195/) (2025)
- [Off-Policy Evaluation Under Nonignorable Missing Data](https://mlsanity.com/paper/2507.06961/) (2025)
- [A Graphical Approach to State Variable Selection in Off-policy Learning](https://mlsanity.com/paper/2501.00854/) (2025)
- [Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approach](https://mlsanity.com/paper/2503.00206/) (2025)
- [A Validation Tool for Designing Reinforcement Learning Environments](https://mlsanity.com/paper/2112.05519/) (2021)
- [Deep Spectral Q-learning with Application to Mobile Health](https://mlsanity.com/paper/2301.00927/) (2023)
- [Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selection](https://mlsanity.com/paper/2301.12553/) (2023)
- [AI in Pharma for Personalized Sequential Decision-Making: Methods, Applications and Opportunities](https://mlsanity.com/paper/2311.18725/) (2023)
- [Capturing reduced-order quantum many-body dynamics out of equilibrium via neural ordinary differential equations](https://mlsanity.com/paper/2512.13913/) (2025)
- [Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learning](https://mlsanity.com/paper/2603.27389/) (2026)
- [From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL](https://mlsanity.com/paper/2605.07057/) (2026)

## Abstract (arXiv)

The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes and partially observable MDPs. We apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.