mlsanityData to 7 Oct 2026

Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates

arXiv 2602.10430 ↗First posted 11 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

24 of 35 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
12 more
  1. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  4. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  5. 2018Remember and Forget for Experience Replaycited by 42
  6. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  7. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  8. 2019Optimistic Distributionally Robust Optimization for Nonparametric Likelihood Approximationcited by 12
  9. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  10. 2020Optimistic Distributionally Robust Policy Optimizationcited by 5
  11. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
  12. 2025Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Bettercited by 3

Built on by

Not cited yet.