mlsanityData to 14 Oct 2026

Value-Guided Repulsion for Offline Reinforcement Learning

arXiv 2610.20190 ↗First posted 14 Oct 2026cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL also fits: Offline RL algorithms

Abstract (arXiv)

Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 34 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

31 of 34 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning3 Dec
  2. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  3. 2025Flow Q-Learning4 Feb
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  5. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  6. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  8. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  10. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 19

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
7 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2021Implicit Behavioral Cloningcited by 452
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  4. 2026Generative Modeling via Driftingcited by 116
  5. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  6. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  7. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  2. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  3. 2024Strategically Conservative Q-Learning
  4. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  5. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  7. 2026SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
  8. 2026Drift Q-Learning