mlsanityData to 7 Oct 2026

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

arXiv 2210.09151 ↗First posted 17 Oct 2022cs.LG cs.AI cs.CLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amount of feedback required by half and improves overall reward recovery. Additionally, we demonstrate that using an abstract state space for the computation of the priors further improves the reward learning and the agent's performance.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 17 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

12 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  5. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2020Quantifying Differences in Reward Functionscited by 62
  8. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  9. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 1; 2025: 2.

8 papers.

  1. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  2. 2024Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?cited by 22
  3. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learningcited by 4
  4. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  5. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferencescited by 2
  6. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learningcited by 2
  7. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learningcited by 1
  8. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hindsight PRIORs for Reward Learning from Human Preferences
  2. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  3. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  4. 2023On The Fragility of Learned Reward Functions
  5. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
  6. 2025Fusing Rewards and Preferences in Reinforcement Learning
  7. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedback
  8. 2026Learning Kernel-Based MDPs from Episodic Preferential Feedback