Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
Not on the map. Why?
Abstract (arXiv)
We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of Qπ, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
22 of 33 references.
References · 22
- 2019Introduction to Online Convex Optimizationcited by 803
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2019Batch Policy Learning under Constraintscited by 244
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
- 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
- 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
- 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
- 2020Batch Value-function Approximation with Only Realizabilitycited by 104
- 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
- 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
10 more
- 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
- 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
- 2019Reinforcement Learning with Convex Constraintscited by 60
- 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
- 2017Balanced Policy Evaluation and Learningcited by 52
- 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 18
- 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
- 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
- 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
- 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Minimal-Assumption Analysis of Q-Learning with Time-Varying Policies
- 2026Sparse Additive Off-Policy Evaluation for Reinforcement Learning with Potentially Limited Number of Trajectories
- 2024Off-policy Distributional Q(λ): Distributional RL without Importance Sampling
- 2023Offline Reinforcement Learning with Additional Covering Distributions
- 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
- 2025Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces
- 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximation
- 2025Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear qπ-Realizability and Concentrability