mlsanityData to 7 Oct 2026

Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

arXiv 2408.04190 ↗First posted 8 Aug 2024cs.LG cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In Reinforcement Learning (RL), designing precise reward functions remains to be a challenge, particularly when aligning with human intent. Preference-based RL (PbRL) was introduced to address this problem by learning reward models from human feedback. However, existing PbRL methods have limitations as they often overlook the second-order preference that indicates the relative strength of preference. In this paper, we propose Listwise Reward Estimation (LiRE), a novel approach for offline PbRL that leverages second-order preference information by constructing a Ranked List of Trajectories (RLT), which can be efficiently built by using the same ternary feedback type as traditional methods. To validate the effectiveness of LiRE, we propose a new offline PbRL dataset that objectively reflects the effect of the estimated rewards. Our extensive experiments on the dataset demonstrate the superiority of LiRE, i.e., outperforming state-of-the-art baselines even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise. Our code is available at https://github.com/chwoong/LiRE

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 61 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

44 of 61 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  7. 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  9. 2023On The Fragility of Learned Reward Functions9 Jan
  10. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  11. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  12. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
4 more
  1. 2021Learning Multimodal Rewards from Rankings27 Sep
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018DeepMind Control Suitecited by 935
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
16 more
  1. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  2. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  3. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  4. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  5. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  8. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  9. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  10. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  11. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  12. 2020PiRank: Scalable Learning To Rank via Differentiable Sortingcited by 26
  13. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  14. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  15. 2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14
  16. 2023Rating-based Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  2. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  3. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  4. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  5. 2024Holistic Utility Preference Learning for Listwise Alignmentcited by 2
  6. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  7. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  8. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  9. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  10. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  11. 2026The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learningcited by 0
  12. 2026Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitationcited by 0
2 more
  1. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  2. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
  3. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  4. 2024Preference Elicitation for Offline Reinforcement Learning
  5. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  6. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
  7. 2025Best Policy Learning from Trajectory Preference Feedback
  8. 2025Offline Safe Policy Optimization From Heterogeneous Feedback