mlsanityData to 7 Oct 2026

A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback

arXiv 2405.12421 ↗First posted 20 May 2024cs.LG cs.AI stat.MLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Inverse Reinforcement Learning (IRL) and Reinforcement Learning from Human Feedback (RLHF) are pivotal methodologies in reward learning, which involve inferring and shaping the underlying reward function of sequential decision-making problems based on observed human demonstrations and feedback. Most prior work in reward learning has relied on prior knowledge or assumptions about decision or preference models, potentially leading to robustness issues. In response, this paper introduces a novel linear programming (LP) framework tailored for offline reward learning. Utilizing pre-collected trajectories without online exploration, this framework estimates a feasible reward set from the primal-dual optimality conditions of a suitably designed LP, and offers an optimality guarantee with provable sample efficiency. Our LP framework also enables aligning the reward functions with human feedback, such as pairwise trajectory comparison data, while maintaining computational tractability and sample efficiency. We demonstrate that our framework potentially achieves better performance compared to the conventional maximum likelihood estimation (MLE) approach through analytical examples and numerical experiments.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 47 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

35 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  2. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  3. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  5. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  10. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
15 more
  1. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  2. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  4. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  5. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  8. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  9. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  10. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  11. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  12. 2022A Ranking Game for Imitation Learningcited by 18
  13. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  14. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraintscited by 15
  15. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 3
  2. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  3. 2026Specifying Reward Functions for RL Without Environment Samplingcited by 0
  4. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  2. 2025Reward Compatibility: A Framework for Inverse RL
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  4. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  5. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  6. 2025Reinforcement Learning from Human Feedback
  7. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  8. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback