mlsanityData to 7 Oct 2026

Policy Optimization via Importance Sampling

arXiv 1809.06098 ↗First posted 17 Sep 2018cs.LG cs.AI stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 10; 2023: 5; 2024: 9; 2025: 7; 2026: 9.

47 papers.

Sort:

By area:

  1. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  2. 2023Stable Target Field for Reduced Variance Score Estimation in Diffusion Modelscited by 44
  3. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 37
  4. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 30
  5. 2022Optimality Guarantees for Particle Belief Approximation of POMDPscited by 20
  6. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  7. 2021The Trade-offs of Domain Adaptation for Neural Language Modelscited by 14
  8. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  9. 2022BR-SNIS: Bias Reduced Self-Normalized Importance Samplingcited by 12
  10. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  11. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  12. 2025Anchored Supervised Fine-Tuningcited by 11
35 more
  1. 2022General Policy Evaluation and Improvement by Learning to Identify Few But Crucial Statescited by 10
  2. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 10
  3. 2024Policy Gradient with Active Importance Samplingcited by 9
  4. 2021Bayesian brains and the R'enyi divergencecited by 6
  5. 2022Reward-Free Policy Space Compression for Reinforcement Learningcited by 6
  6. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  7. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  8. 2021Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluationcited by 5
  9. 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularizationcited by 5
  10. 2022Low Variance Off-policy Evaluation with State-based Importance Samplingcited by 5
  11. 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Ratecited by 5
  12. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  13. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  14. 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
  15. 2023Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimizationcited by 3
  16. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  17. 2023Parallel bootstrap-based on-policy deep reinforcement learning for continuous flow control applicationscited by 2
  18. 2024Highway Reinforcement Learningcited by 2
  19. 2024Automated Off-Policy Estimator Selection via Supervised Learningcited by 2
  20. 2024Compatible Gradient Approximations for Actor-Critic Algorithmscited by 2
  21. 2023Truncating Trajectories in Monte Carlo Reinforcement Learningcited by 1
  22. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  23. 2025Reusing Trajectories in Policy Gradients Enables Fast Convergencecited by 1
  24. 2026GIPO: Gaussian Importance Sampling Policy Optimizationcited by 1
  25. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matchingcited by 1
  26. 2022Offline Policy Optimization in RL with Variance Regularizatoncited by 0
  27. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  28. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policiescited by 0
  29. 2026Variance Reduction Based Experience Replay for Policy Optimizationcited by 0
  30. 2026Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimizationcited by 0
  31. 2026Robust Policy Optimization via Adversarial Importance Samplingcited by 0
  32. 2026Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct Itcited by 0
  33. 2026ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learningcited by 0
  34. 2026Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reusecited by 0
  35. 2026Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?cited by 0