mlsanityData to 14 Oct 2026

Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method

arXiv 2203.04951 ↗First posted 9 Mar 2022cs.RO cs.AI cs.HCcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

For robots to be effectively deployed in novel environments and tasks, they must be able to understand the feedback expressed by humans during intervention. This can either correct undesirable behavior or indicate additional preferences. Existing methods either require repeated episodes of interactions or assume prior known reward features, which is data-inefficient and can hardly transfer to new tasks. We relax these assumptions by describing human tasks in terms of object-centric sub-tasks and interpreting physical interventions in relation to specific objects. Our method, Object Preference Adaptation (OPA), is composed of two key stages: 1) pre-training a base policy to produce a wide variety of behaviors, and 2) online-updating according to human feedback. The key to our fast, yet simple adaptation is that general interaction dynamics between agents and objects are fixed, and only object-specific preferences are updated. Our adaptation occurs online, requires only one human intervention (one-shot), and produces new behaviors never seen during training. Trained on cheap synthetic data instead of expensive human demonstrations, our policy correctly adapts to human perturbations on realistic tasks on a physical 7DOF robot. Videos, code, and supplementary material are provided.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 46 references that are in the map. First posted on arXiv 9 Mar 2022.

Builds on

27 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 26

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  2. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  3. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  4. 2019On the Convergence of Adam and Beyondcited by 841
  5. 2016Learning to learn by gradient descent by gradient descentcited by 620
  6. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  10. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  11. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  12. 2018Graph networks as learnable physics engines for inference and controlcited by 244
14 more
  1. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
  2. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  3. 2019Graph-Structured Visual Imitationcited by 43
  4. 2020Multi-Loss Weighting with Coefficient of Variationscited by 36
  5. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  6. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  7. 2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
  8. 2020Feature Expansive Reward Learning: Rethinking Human Inputcited by 25
  9. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrationscited by 19
  10. 2020Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulationcited by 17
  11. 2022Inducing Structure in Reward Learning by Learning Featurescited by 16
  12. 2018Including Uncertainty when Learning from Human Correctionscited by 15
  13. 2018Learning under Misspecified Objective Spacescited by 14
  14. 2020Visual Relationship Detection using Scene Graphs: A Surveycited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023GUARD: A Safe Reinforcement Learning Benchmarkcited by 13
  2. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
  2. 2023Active Reward Learning from Online Preferences
  3. 2024Adaptive Querying for Reward Learning from Human Feedback
  4. 2026Adaptation of Generalist Robot Policies with Minimal Data
  5. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  6. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences
  7. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  8. 2026Human Preference Modeling Using Visual Motion Prediction Improves Robot Skill Learning from Egocentric Human Video ↗