mlsanityData to 7 Oct 2026

Minimax Weight and Q-Function Learning for Off-Policy Evaluation

arXiv 1910.12809 ↗First posted 28 Oct 2019cs.LG stat.MLcited by 150

Not on the map. Why?

Cited by 57 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions. Our contributions include: (1) A new estimator, MWL, that directly estimates importance ratios over the state-action distributions, removing the reliance on knowledge of the behavior policy as in prior work (Liu et al., 2018). (2) Another new estimator, MQL, obtained by swapping the roles of importance weights and value-functions in MWL. MQL has an intuitive interpretation of minimizing average Bellman errors and can be combined with MWL in a doubly robust manner. (3) Several additional results that offer further insights into these methods, including the sample complexity analyses of MWL and MQL, their asymptotic optimality in the tabular setting, how the learned importance weights depend the choice of the discriminator class, and how our methods provide a unified view of some old and new algorithms in RL.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 7; 2021: 37; 2022: 34; 2023: 27; 2024: 15; 2025: 17; 2026: 13.

150 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  5. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  6. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  7. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  9. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  10. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
38 more
  1. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 43
  2. 2021Universal Off-Policy Evaluationcited by 43
  3. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  4. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  6. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  7. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  8. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  9. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  10. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  11. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  12. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  13. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  14. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  15. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  16. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  17. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  18. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  19. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  20. 2021Active Offline Policy Selectioncited by 26
  21. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  22. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  23. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  24. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  25. 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparisoncited by 22
  26. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  27. 2022Markovian Interference in Experimentscited by 22
  28. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  29. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  30. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  31. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  32. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  33. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  34. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  35. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  36. 2021Minimax Model Learningcited by 16
  37. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  38. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14

50 most cited of 150 All 150 (JSON)