mlsanityData to 14 Oct 2026

SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling

arXiv 2608.02951 ↗First posted 3 Aug 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) for general stochastic MDPs often requires training a reward model. Existing reward-model-free methods are either restricted to bandits or deterministic MDPs, such as DPO or P3O, or use zeroth-order, gradient-free optimization, which in general exhibits a slower convergence rate than gradient-based algorithms. Furthermore, existing reward-model-free preference-based RL algorithms almost exclusively use trajectory-level feedback, which can require significant effort from a human evaluator when trajectories are long. On the other hand, segments are much shorter, so they are easier to compare and evaluate. In this paper, we introduce a novel reward-model-free, critic-free, and gradient-based PbRL algorithm compatible with segment preferences named Segment Pairwise Proximal Policy Optimization (SP3O). SP3O utilizes segment-level preference feedback to construct an accurate policy value difference estimator via off-policy importance sampling, and then uses the estimator to compute the policy gradient via a PPO-type loss function. We provide a theoretical basis for the algorithm and analyze the tradeoff in choosing the segment length. We also evaluate it experimentally against other PbRL/RLHF algorithms in robotic control and LLM finetuning settings to show its improved performance, especially in long-horizon tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 44 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

26 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference25 Sep
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Models of human preference for learning reward functions5 Jun
  7. 2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning30 May
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  10. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  9. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  11. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  12. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
3 more
  1. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  2. 2022Making Reinforcement Learning Work on Swimmercited by 6
  3. 2025Reinforcement Learning with Segment Feedbackcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Direct Preference-based Policy Optimization without Reward Modeling
  2. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  3. 2024Advances in Preference-based Reinforcement Learning: A Review
  4. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  5. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences
  6. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  7. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  8. 2025Best Policy Learning from Trajectory Preference Feedback