mlsanityData to 14 Oct 2026

PAWS: Preference Learning with Advantage-Weighted Segments

arXiv 2606.11982 ↗First posted 10 Jun 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) learns policies from human trajectory-level comparisons, avoiding explicit reward design and expert demonstrations. Existing methods typically train utility functions on trajectory or segment-level preferences while relying on per-step utility estimates during policy optimization. This training and inference mismatch induces a distribution shift that severely degrades temporal credit assignment and limits policy learning. We analyze this issue and propose PAWS, a segment-based preference learning method that performs policy updates directly using segment-level advantage functions. By aligning utility training with policy optimization, PAWS preserves trajectory-level preference information and avoids unreliable per-step learning signals. Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches, highlighting the importance of distribution-consistent preference learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 49 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

35 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers20 Sep

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2021Convex optimizationcited by 2,515
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
22 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2018Maximum a Posteriori Policy Optimisationcited by 328
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  11. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  12. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  13. 2022Models of human preference for learning reward functionscited by 54
  14. 2021Learning Multimodal Rewards from Rankingscited by 41
  15. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  16. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 25
  17. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
  18. 2024Learning from negative feedback, or positive feedback or bothcited by 13
  19. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  20. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  21. 2025Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objectscited by 10
  22. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  2. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition
  3. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  4. 2026MAPL: Multi-Objective Preference Learning for Robot Locomotion ↗
  5. 2026Freeform Preference Learning for Robotic Manipulation
  6. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  7. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  8. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning