mlsanityData to 7 Oct 2026

Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations

arXiv 2408.05610 ↗First posted 10 Aug 2024cs.RO cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We study the problem of cross-embodiment inverse reinforcement learning, where we wish to learn a reward function from video demonstrations in one or more embodiments and then transfer the learned reward to a different embodiment (e.g., different action space, dynamics, size, shape, etc.). Learning reward functions that transfer across embodiments is important in settings such as teaching a robot a policy via human video demonstrations or teaching a robot to imitate a policy from another robot with a different embodiment. However, prior work has only focused on cases where near-optimal demonstrations are available, which is often difficult to ensure. By contrast, we study the setting of cross-embodiment reward learning from mixed-quality demonstrations. We demonstrate that prior work struggles to learn generalizable reward representations when learning from mixed-quality data. We then analyze several techniques that leverage human feedback for representation learning and alignment to enable effective cross-embodiment learning. Our results give insight into how different representation learning techniques lead to qualitatively different reward shaping behaviors and the importance of human feedback when learning from mixed-quality, mixed-embodiment data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 63 references that are in the map. First posted on arXiv 10 Aug 2024.

Builds on

42 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learning11 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  4. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types23 Aug
  5. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  6. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  7. 2021Learning Multimodal Rewards from Rankings27 Sep
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  7. 2018Behavioral Cloning from Observationcited by 482
  8. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 219
  11. 2018Generative Adversarial Imitation from Observationcited by 183
  12. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
21 more
  1. 2019Temporal Cycle-Consistency Learningcited by 151
  2. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  3. 2019Recent Advances in Imitation Learning from Observationcited by 92
  4. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  5. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  6. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  7. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  8. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  9. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  10. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  11. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  12. 2021Human irrationality: both bad and good for reward inferencecited by 25
  13. 2021Learning Reward Functions from Scale Feedbackcited by 24
  14. 2020Active Preference Learning using Maximum Regretcited by 21
  15. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  16. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Videocited by 16
  17. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  18. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  19. 2023SIRL: Similarity-based Implicit Representation Learningcited by 13
  20. 2023Leveraging Human Feedback to Evolve and Discover Novel Emergent Behaviors in Robot Swarmscited by 4
  21. 2024Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocauterycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Embodiment Transfer via Behavior-Aligned Representations
  2. 2024Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigation
  3. 2022Graph Inverse Reinforcement Learning from Diverse Videos
  4. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
  5. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
  6. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learning
  7. 2026CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
  8. 2025NIL: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Models