mlsanityData to 13 Oct 2026

Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards

arXiv 2410.23289 ↗First posted 30 Oct 2024cs.RO cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Training robots directly from human videos is an emerging area in robotics and computer vision. While there has been notable progress with two-fingered grippers, learning autonomous tasks for multi-fingered robot hands in this way remains challenging. A key reason for this difficulty is that a policy trained on human hands may not directly transfer to a robot hand due to morphology differences. In this work, we present HuDOR, a technique that enables online fine-tuning of policies by directly computing rewards from human videos. Importantly, this reward function is built using object-oriented trajectories derived from off-the-shelf point trackers, providing meaningful learning signals despite the morphology gap and visual differences between human and robot hands. Given a single video of a human solving a task, such as gently opening a music box, HuDOR enables our four-fingered Allegro hand to learn the task with just an hour of online interaction. Our experiments across four tasks show that HuDOR achieves a 4x improvement over baselines. Code and videos are available on our website, https://object-rewards.github.io.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Builds on 45 of its 50 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

45 of 50 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2023Segment Anythingcited by 8,195
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,231
  5. 2015Continuous control with deep reinforcement learningcited by 3,290
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,760
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,512
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 857
  10. 2019Solving Rubik's Cube with a Robot Handcited by 702
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 549
  12. 2023CoTracker: It is Better to Track Togethercited by 518
33 more
  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 482
  2. 2023Learning Interactive Real-World Simulatorscited by 445
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 361
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 222
  5. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 216
  6. 2024Behavior Generation with Latent Actionscited by 159
  7. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 150
  8. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  9. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 137
  10. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 106
  11. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  12. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  13. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  14. 2017Intel RealSense Stereoscopic Depth Camerascited by 82
  15. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  16. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  17. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  18. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  19. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  20. 2024OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulationcited by 67
  21. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  22. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 63
  23. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  24. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  25. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  26. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  27. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  28. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  29. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  30. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  31. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  32. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  33. 2019Convex Optimisation for Inverse Kinematicscited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 15; 2026: 10.

25 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 656
  2. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 79
  3. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  4. 2025EgoZero: Robot Learning from Smart Glassescited by 31
  5. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 28
  6. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  7. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 25
  8. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 23
  9. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  10. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  11. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 16
  12. 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 13
13 more
  1. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 11
  2. 2025RUKA: Rethinking the Design of Humanoid Hands with Learningcited by 8
  3. 2025Feel the Force: Contact-Driven Learning from Humanscited by 8
  4. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 8
  5. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 8
  6. 2026LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisitioncited by 4
  7. 2026Human Universal Graspingcited by 4
  8. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  9. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 1
  10. 2026Ruka-v2: Tendon Driven Open-Source Dexterous Hand with Wrist and Abduction for Robot Learningcited by 1
  11. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  12. 2026ContactExplorer: Contact Coverage-Guided Exploration for General-Purpose Dexterous Manipulationcited by 0
  13. 2026X-Reset: Scaling Object-Centric Reinforcement Learning via Cross-Embodiment Resetscited by 0