mlsanityData to 7 Oct 2026

A State Augmentation based approach to Reinforcement Learning from Human Preferences

arXiv 2302.08734 ↗First posted 17 Feb 2023cs.AI cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning has suffered from poor reward specification, and issues for reward hacking even in simple enough domains. Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs by a human in the loop indicating their preferences about the agent's behavior to learn a reward model. In this work, we present a state augmentation technique that allows the agent's reward model to be robust and follow an invariance consistency that significantly improved performance, i.e. the reward recovery and subsequent return computed using the learned policy over our baseline PEBBLE. We validate our method on three domains, Mountain Car, a locomotion task of Quadruped-Walk, and a robotic manipulation task of Sweep-Into, and find that using the proposed augmentation the agent not only benefits in the overall performance but does so, quite early in the agent's training phase.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 37 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

24 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences17 Oct
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  3. 2016OpenAI Gymcited by 1,684
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  8. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  9. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  11. 2017Inverse Reward Designcited by 246
  12. 2017Visualizing and Understanding Atari Agentscited by 141
8 more
  1. 2021On the Expressivity of Markov Rewardcited by 71
  2. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  3. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  4. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  5. 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
  6. 2019Making Smart Homes Smarter: Optimizing Energy Consumption with Human in the Loopcited by 3
  7. 2021A Novel Framework for Neural Architecture Search in the Hill Climbing Domaincited by 2
  8. 2020Fine-grained Language Identification with Multilingual CapsNet Modelcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0
  2. 2025State-Augmented Graphs for Circular Economy Triagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning
  2. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  3. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  4. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  5. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  6. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  7. 2023REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback
  8. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning