# Batch Stationary Distribution Estimation

arXiv: https://arxiv.org/abs/2003.00722 · HTML: https://mlsanity.com/paper/2003.00722/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 19 of 19)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [GFlowNet Foundations](https://mlsanity.com/paper/2111.09266/) (2021)
- [Benchmarks for Deep Off-Policy Evaluation](https://mlsanity.com/paper/2103.16596/) (2021)
- [Provably Good Batch Reinforcement Learning Without Great Exploration](https://mlsanity.com/paper/2007.08202/) (2020)
- [Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization](https://mlsanity.com/paper/2104.13877/) (2021)
- [One-Step Flow Policy Mirror Descent](https://mlsanity.com/paper/2507.23675/) (2025)
- [A Comprehensive Survey on Rare Event Prediction](https://mlsanity.com/paper/2309.11356/) (2023)
- [Continual Learning In Environments With Polynomial Mixing Times](https://mlsanity.com/paper/2112.07066/) (2021)
- [A Unified Framework for Alternating Offline Model Training and Policy Learning](https://mlsanity.com/paper/2210.05922/) (2022)
- [Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds](https://mlsanity.com/paper/2103.05741/) (2021)
- [Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction](https://mlsanity.com/paper/2303.12534/) (2023)
- [Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction](https://mlsanity.com/paper/2212.07486/) (2022)
- [Variational Latent Branching Model for Off-Policy Evaluation](https://mlsanity.com/paper/2301.12056/) (2023)
- [Nonparametric Density Estimation under Distribution Drift](https://mlsanity.com/paper/2302.02460/) (2023)
- [Trajectory World Models for Heterogeneous Environments](https://mlsanity.com/paper/2502.01366/) (2025)
- [Model-based Offline Policy Optimization with Adversarial Network](https://mlsanity.com/paper/2309.02157/) (2023)
- [Forward-Free Diffusion Language Models with BPTT-Free Looped Refinement](https://mlsanity.com/paper/2606.08357/) (2026)
- [Scalable Offline Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2410.17898/) (2024)
- [Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/2107.05798/) (2021)

## Abstract (arXiv)

We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a fixed set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.