mlsanityData to 14 Oct 2026

Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making

arXiv 2002.01751 ↗First posted 5 Feb 2020stat.ML cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes and partially observable MDPs. We apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 7; 2024: 4; 2025: 6; 2026: 2.

30 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  3. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  4. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  5. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  6. 2024A Review of Reinforcement Learning in Financial Applicationscited by 13
  7. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  8. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  9. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  10. 2023Testing for the Markov Property in Time Series via Deep Conditional Generative Learningcited by 9
  11. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  12. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
18 more
  1. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  2. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  3. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  4. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  5. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  6. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  7. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  8. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  9. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2
  10. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1
  11. 2025Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approachcited by 1
  12. 2021A Validation Tool for Designing Reinforcement Learning Environmentscited by 0
  13. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  14. 2023Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selectioncited by 0
  15. 2023AI in Pharma for Personalized Sequential Decision-Making: Methods, Applications and Opportunitiescited by 0
  16. 2025Capturing reduced-order quantum many-body dynamics out of equilibrium via neural ordinary differential equationscited by 0
  17. 2026Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learningcited by 0
  18. 2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RLcited by 0