mlsanityData to 7 Oct 2026

MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention

arXiv 2406.16258 ↗First posted 24 Jun 2024cs.RO cs.AI cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Aligning robot behavior with human preferences is crucial for deploying embodied AI agents in human-centered environments. A promising solution is interactive imitation learning from human intervention, where a human expert observes the policy's execution and provides interventions as feedback. However, existing methods often fail to utilize the prior policy efficiently to facilitate learning, thus hindering sample efficiency. In this work, we introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning), designed for sample-efficient alignment from human intervention. Instead of inferring the complete human behavior characteristics, MEReQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions. It then employs Residual Q-Learning (RQL) to align the policy with human preferences using this residual reward function. Extensive evaluations on simulated and real-world tasks demonstrate that MEReQ achieves sample-efficient policy alignment from human intervention.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 57 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

32 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learning11 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023Active Reward Learning from Online Preferences27 Feb
  4. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback11 Aug
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  7. 2023AI Alignment: A Comprehensive Surveycited by 256
  8. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  9. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  12. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
14 more
  1. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  2. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  3. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  4. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  5. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  6. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  7. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  8. 2020Feature Expansive Reward Learning: Rethinking Human Inputcited by 25
  9. 2021Learning Reward Functions from Scale Feedbackcited by 24
  10. 2020LESS is More: Rethinking Probabilistic Models of Human Behaviorcited by 17
  11. 2020APPLI: Adaptive Planner Parameter Learning From Interventionscited by 16
  12. 2018Learning under Misspecified Objective Spacescited by 14
  13. 2023Residual Q-Learning: Offline and Online Policy Customization without Valuecited by 9
  14. 2019Interactively shaping robot behaviour with unlabeled human instructionscited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Residual Policy Gradient: A Reward View of KL-regularized Objectivecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
  2. 2023Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning
  3. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  4. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  5. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
  7. 2023Coherent Soft Imitation Learning
  8. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment