# Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.02363 · HTML: https://mlsanity.com/paper/1905.02363/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2019.

## Built on by (most cited first, 12 of 12)

- [Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization](https://mlsanity.com/paper/2503.12645/) (2025)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Exclusively Penalized Q-learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.14082/) (2024)
- [Transductive Off-policy Proximal Policy Optimization](https://mlsanity.com/paper/2406.03894/) (2024)
- [Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach](https://mlsanity.com/paper/2208.00755/) (2022)
- [Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies](https://mlsanity.com/paper/2405.18792/) (2024)
- [Augmenting Offline RL with Unlabeled Data](https://mlsanity.com/paper/2406.07117/) (2024)
- [Intelligent Control of Spacecraft Reaction Wheel Attitude Using Deep Reinforcement Learning](https://mlsanity.com/paper/2507.08366/) (2025)
- [Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces](https://mlsanity.com/paper/2508.14102/) (2025)
- [Smooth Gate Functions for Soft Advantage Policy Optimization](https://mlsanity.com/paper/2602.19345/) (2026)
- [Reinforcement Learning to Accelerate Primal-Dual Hybrid Gradient for Linear Programming](https://mlsanity.com/paper/2610.01546/) (2026)

## Abstract (arXiv)

In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it difficult to reuse old samples with large IS weights. In this paper, we consider PPO, a representative on-policy algorithm, and propose its improvement by dimension-wise IS weight clipping which separately clips the IS weight of each action dimension to avoid large bias and adaptively controls the IS weight to bound policy update from the current policy. This new technique enables efficient learning for high action-dimensional tasks and reusing of old samples like in off-policy learning to increase the sample efficiency. Numerical results show that the proposed new algorithm outperforms PPO and other RL algorithms in various Open AI Gym tasks.