mlsanityData to 14 Oct 2026

How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions

arXiv 2504.12284 ↗First posted 16 Apr 2025cs.CV cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We tackle the novel problem of predicting 3D hand motion and contact maps (or Interaction Trajectories) given a single RGB view, action text, and a 3D contact point on the object as input. Our approach consists of (1) Interaction Codebook: a VQVAE model to learn a latent codebook of hand poses and contact points, effectively tokenizing interaction trajectories, (2) Interaction Predictor: a transformer-decoder module to predict the interaction trajectory from test time inputs by using an indexer module to retrieve a latent affordance from the learned codebook. To train our model, we develop a data engine that extracts 3D hand poses and contact trajectories from the diverse HoloAssist dataset. We evaluate our model on a benchmark that is 2.5-10X larger than existing works, in terms of diversity of objects and interactions observed, and test for generalization of the model across object categories, action categories, tasks, and scenes. Experimental results show the effectiveness of our approach over transformer & diffusion baselines across all settings.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

45 of 55 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2022Human Motion Diffusion Modelcited by 1,080
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  9. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  10. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  11. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  12. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
33 more
  1. 2023Reconstructing Hands in 3D with Transformerscited by 306
  2. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  3. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  5. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  6. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  7. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  8. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  9. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  10. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  11. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  12. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  13. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  14. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 86
  15. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  16. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  17. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62
  18. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 62
  19. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 55
  20. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  21. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 53
  22. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  23. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 48
  24. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 44
  25. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  26. 2023Task-Oriented Human-Object Interactions Generation with Implicit Neural Representationscited by 42
  27. 2022Reconstructing Hand-Held Objects from Monocular Videocited by 35
  28. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  29. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 34
  30. 2024Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioningcited by 30
  31. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  32. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  33. 2023Mitigating Perspective Distortion-induced Shape Ambiguity in Image Cropscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  2. 2025Learning Dense Hand Contact Estimation from Imbalanced Datacited by 14
  3. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
  4. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  5. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  6. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  7. 2025Affordance-Guided Diffusion Prior for 3D Hand Reconstructioncited by 2
  8. 2026HOPE: Hand-Object Pressure Estimation from Monocular Videoscited by 2
  9. 2025Bimanual 3D Hand Motion and Articulation Forecasting in Everyday Imagescited by 1
  10. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1