mlsanityData to 7 Oct 2026

Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with Physics

arXiv 2405.02676 ↗First posted 4 May 2024cs.CV cs.GRcited by 6

Not on the map. Why?

Abstract (arXiv)

Hand manipulating objects is an important interaction motion in our daily activities. We faithfully reconstruct this motion with a single RGBD camera by a novel deep reinforcement learning method to leverage physics. Firstly, we propose object compensation control which establishes direct object control to make the network training more stable. Meanwhile, by leveraging the compensation force and torque, we seamlessly upgrade the simple point contact model to a more physical-plausible surface contact model, further improving the reconstruction accuracy and physical correctness. Experiments indicate that without involving any heuristic physical rules, this work still successfully involves physics in the reconstruction of hand-object interactions which are complex motions hard to imitate with deep reinforcement learning. Our code and data are available at https://github.com/hu-hy17/HOIC.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

32 of 67 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  6. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  7. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  8. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  9. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  10. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  11. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  12. 2021A System for General In-Hand Object Re-Orientationcited by 155
20 more
  1. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  2. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  3. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  5. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  6. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  7. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 91
  8. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  9. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  10. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 65
  11. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  12. 2022Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Searchcited by 61
  13. 2020PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Timecited by 50
  14. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49
  15. 2022Learning to Use Chopsticks in Diverse Gripping Stylescited by 29
  16. 2022Learning Diverse and Physically Feasible Dexterous Grasps with Generative Model and Bilevel Optimizationcited by 20
  17. 2022VToonify: Controllable High-Resolution Portrait Video Style Transfercited by 20
  18. 2022Physical Interaction: Reconstructing Hand-object Interactions with Physicscited by 19
  19. 2022Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wildcited by 17
  20. 2015Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone for Rectangular Support Areascited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  2. 2026PAD-Hand: Physics-Aware Diffusion for Hand Motion Recoverycited by 4
  3. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3
  4. 2025Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splattingcited by 0
  5. 2025VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimationcited by 0
  6. 2026DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulationcited by 0