mlsanityData to 14 Oct 2026

Central Limit Theorems for Transition Probabilities of Controlled Markov Chains

arXiv 2508.01517 ↗First posted 2 Aug 2025math.ST math.PR stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

We develop a central limit theorem (CLT) for a non-parametric estimator of the transition matrices in controlled Markov chains (CMCs) with finite state-action spaces. Our results establish precise conditions on the logging policy under which the estimator is asymptotically normal, and reveal settings in which no CLT can exist. We then build on it to derive CLTs for the value, Q-, and advantage functions of any stationary stochastic policy, including the optimal policy recovered from the estimated model. Goodness-of-fit tests are derived as a corollary, which enable to test whether the logged data is stochastic. These results provide new statistical tools for offline policy evaluation and optimal policy recovery, and enable hypothesis tests for transition probabilities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 64 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

19 of 64 references.

References · 19

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  5. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  8. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  9. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  10. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 80
  11. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  12. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
7 more
  1. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  2. 2019Estimating the Mixing Time of Ergodic Markov Chainscited by 25
  3. 2020Probabilistic Machine Learning for Healthcarecited by 23
  4. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  5. 2019Uncertainty Quantification and Exploration for Reinforcement Learningcited by 11
  6. 2022Offline Estimation of Controlled Markov Chains: Minimaxity and Sample Complexitycited by 5
  7. 2023Online Stabilization of Unknown Linear Time-Varying Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Model-based Bootstrap of Controlled Markov Chainscited by 0