Data to 14 Oct 2026

Inverse Preference Learning: Preference-based RL without a Reward Function

arXiv 2305.15363 ↗First posted 24 May 2023cs.LGcited by 68

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods na"ively combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the Q-function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 61 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

41 of 61 references.

Closest ancestors · 8

Same direction, newest first

Other references · 33

2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2020Learning to summarize from human feedbackcited by 2,588
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2016Concrete Problems in AI Safetycited by 1,642
2016Generative Adversarial Imitation Learningcited by 1,597
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782

21 more

2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2021Recursively Summarizing Books with Human Feedbackcited by 290
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
2017Inverse Reward Designcited by 249
2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
2022Inferring Rewards from Language in Contextcited by 50
2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 29
2019Controlling Assistive Robots with Learned Latent Actionscited by 28
2016Visual Closed-Loop Control for Pouring Liquidscited by 26
2021Offline Preference-Based Apprenticeship Learningcited by 17
2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 16
2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 20; 2025: 25; 2026: 11.

68 papers.

Sort:

By area:

2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 33
2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 26
2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 19
2024Optimal Design for Human Preference Elicitationcited by 17
2024Aligning Human Motion Generation with Human Perceptionscited by 16
2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 13

38 more

2023Selective Sampling and Imitation Learning via Online Regressioncited by 12
2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
2025Preference Optimization for Combinatorial Optimization Problemscited by 11
2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigationcited by 7
2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 6
2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learningcited by 3
2024Preference Elicitation for Offline Reinforcement Learningcited by 3
2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
2025Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Modelscited by 3
2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
2025Training LLM Agents to Empower Humanscited by 3
2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
2025Residual Policy Gradient: A Reward View of KL-regularized Objectivecited by 2
2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
2025Reinforcement Learning from Adversarial Preferences in Tabular MDPscited by 2
2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
2025Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemmacited by 2
2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2
2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learningcited by 1
2025Model-Based Exploration in Monitored Markov Decision Processescited by 1
2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
2025Offline Safe Policy Optimization From Heterogeneous Feedbackcited by 1
2026Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Carecited by 1
2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
2026Freeform Preference Learning for Robotic Manipulationcited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way