# Importance Resampling for Off-policy Prediction

arXiv: https://arxiv.org/abs/1906.04328 · HTML: https://mlsanity.com/paper/1906.04328/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 26 of 26)

- [PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation](https://mlsanity.com/paper/2509.19128/) (2025)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation](https://mlsanity.com/paper/2405.17066/) (2024)
- [Conformal Off-policy Prediction](https://mlsanity.com/paper/2206.06711/) (2022)
- [OVD: On-policy Verbal Distillation](https://mlsanity.com/paper/2601.21968/) (2026)
- [AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2311.02194/) (2023)
- [Revisiting Prioritized Experience Replay: A Value Perspective](https://mlsanity.com/paper/2102.03261/) (2021)
- [SOPE: Spectrum of Off-Policy Estimators](https://mlsanity.com/paper/2111.03936/) (2021)
- [Bayesian Off-Policy Evaluation and Learning for Large Action Spaces](https://mlsanity.com/paper/2402.14664/) (2024)
- [Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning](https://mlsanity.com/paper/2412.05783/) (2024)
- [Rankitect: Ranking Architecture Search Battling World-class Engineers at Meta Scale](https://mlsanity.com/paper/2311.08430/) (2023)
- [Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies](https://mlsanity.com/paper/2405.18792/) (2024)
- [Learning robust driving policies without online exploration](https://mlsanity.com/paper/2103.08070/) (2021)
- [Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling](https://mlsanity.com/paper/2106.10566/) (2021)
- [K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control](https://mlsanity.com/paper/2306.04836/) (2023)
- [Variance Reduction based Partial Trajectory Reuse to Accelerate Policy Gradient Optimization](https://mlsanity.com/paper/2205.02976/) (2022)
- [Variance Reduction based Experience Replay for Policy Optimization](https://mlsanity.com/paper/2208.12341/) (2022)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Variational Actor-Critic Algorithms](https://mlsanity.com/paper/2108.01215/) (2021)
- [Variance Reduction Based Experience Replay for Policy Optimization](https://mlsanity.com/paper/2602.05379/) (2026)
- [Deep Doubly Debiased Longitudinal Effect Estimation with ICE G-Computation](https://mlsanity.com/paper/2602.12379/) (2026)
- [BadDLM: Backdooring Diffusion Language Models with Diverse Targets](https://mlsanity.com/paper/2605.09397/) (2026)
- [Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings](https://mlsanity.com/paper/2605.14284/) (2026)
- [Learning Implicit Causal World Models from Multi-Agent Demonstrations](https://mlsanity.com/paper/2607.26336/) (2026)
- [Gated Q-learning: Add Off-Policy Bias to Taste](https://mlsanity.com/paper/2607.28916/) (2026)

## Abstract (arXiv)

Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.