mlsanityData to 14 Oct 2026

From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

arXiv 2606.01123 ↗First posted 31 May 2026cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 75 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

59 of 75 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning28 Sep
  2. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning12 Dec
  3. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learning8 Aug
  4. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  5. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback4 Feb
  6. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
  8. 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  10. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  11. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  12. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
2 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2018DeepMind Control Suitecited by 956
  10. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 561
  11. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  12. 2021Reinforcement learning based recommender systems: A surveycited by 188
33 more
  1. 2020Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learningcited by 140
  2. 2021Learning One Representation to Optimize All Rewardscited by 115
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2018Universal Successor Features Approximatorscited by 103
  5. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  6. 2024Foundation Policies with Hilbert Representationscited by 94
  7. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  8. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  9. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  10. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  13. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  14. 2025Temporal Difference Flowscited by 24
  15. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
  16. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  17. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  18. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  19. 2023PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clippingcited by 17
  20. 2024A Distributional Analogue to the Successor Representationcited by 17
  21. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  22. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  23. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  24. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  25. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  26. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  27. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  28. 2025Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learningcited by 4
  29. 2025Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantizationcited by 3
  30. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  31. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  32. 2026Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Modelscited by 2
  33. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Task Inference Beyond Least Squares in Behavioral Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  2. 2024Online Policy Learning from Offline Preferences
  3. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  4. 2024Advances in Preference-based Reinforcement Learning: A Review
  5. 2024Preference Elicitation for Offline Reinforcement Learning
  6. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  7. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration
  8. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency