mlsanityData to 14 Oct 2026

GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping

arXiv 2112.11454 ↗First posted 21 Dec 2021cs.CVcited by 144

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating digital humans that move realistically has many applications and is widely studied, but existing methods focus on the major limbs of the body, ignoring the hands and head. Hands have been separately studied, but the focus has been on generating realistic static grasps of objects. To synthesize virtual characters that interact with the world, we need to generate full-body motions and realistic hand grasps simultaneously. Both sub-problems are challenging on their own and, together, the state-space of poses is significantly larger, the scales of hand and body motions differ, and the whole-body posture and the hand grasp must agree, satisfy physical constraints, and be plausible. Additionally, the head is involved because the avatar must look at the object to interact with it. For the first time, we address the problem of generating full-body, hand and head motions of an avatar grasping an unknown object. As input, our method, called GOAL, takes a 3D object, its position, and a starting 3D body pose and shape. GOAL outputs a sequence of whole-body poses using two novel networks. First, GNet generates a goal whole-body grasp with a realistic body, head, arm, and hand pose, as well as hand-object contact. Second, MNet generates the motion between the starting and goal pose. This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact. To achieve this, the networks exploit a representation that combines SMPL-X body parameters and 3D vertex offsets. We train and evaluate GOAL, both qualitatively and quantitatively, on the GRAB dataset. Results show that GOAL generalizes well to unseen objects, outperforming baselines. GOAL takes a step towards synthesizing realistic full-body object grasping.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Builds on 34 of its 77 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

34 of 77 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  5. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  6. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  7. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
  8. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  9. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  10. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  11. 2017On human motion prediction using recurrent neural networkscited by 338
  12. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
22 more
  1. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 246
  2. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  3. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 185
  4. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  5. 2020History Repeats Itself: Human Motion Prediction via Motion Attentioncited by 165
  6. 2021A System for General In-Hand Object Re-Orientationcited by 165
  7. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  8. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  9. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  10. 2019Generating 3D People in Scenes without Peoplecited by 127
  11. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  12. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  13. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  14. 2018A Neural Temporal Model for Human Motion Predictioncited by 87
  15. 2019Context-aware Human Motion Predictioncited by 84
  16. 2021THUNDR: Transformer-based 3D HUmaN Reconstruction with Markerscited by 50
  17. 2018Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamiccited by 41
  18. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 40
  19. 2020Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learningcited by 37
  20. 2016Understanding and Exploiting Object Interaction Landscapescited by 8
  21. 2020Pose to Seat: Automated Design of Body-Supporting Surfacescited by 5
  22. 2015A Whole-Body Pose Taxonomy for Loco-Manipulation Taskscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 28; 2024: 46; 2025: 30; 2026: 24.

144 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  2. 2022EDGE: Editable Dance Generation From Musiccited by 287
  3. 2023Object Motion Guided Human Motion Synthesiscited by 212
  4. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  5. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  6. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
  7. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 137
  8. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  9. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  10. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  11. 2023Controllable Human-Object Interaction Synthesiscited by 109
  12. 2023Human Motion Generation: A Surveycited by 96
38 more
  1. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  2. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 93
  3. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  4. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  5. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  6. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  7. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  8. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  9. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  10. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  11. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  12. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  13. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  14. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 62
  15. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 60
  16. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
  17. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  18. 2022GenDexGrasp: Generalizable Dexterous Graspingcited by 56
  19. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  20. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 55
  21. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  22. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 53
  23. 2024ReGenNet: Towards Human Action-Reaction Synthesiscited by 53
  24. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  25. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  26. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
  27. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  28. 2022FLEX: Full-Body Grasping Without Full-Body Graspscited by 46
  29. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  30. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  31. 2023Generative Proxemics: A Prior for 3D Social Interaction from Imagescited by 43
  32. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Modelscited by 43
  33. 2023Task-Oriented Human-Object Interactions Generation with Implicit Neural Representationscited by 42
  34. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearancescited by 42
  35. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  36. 2023PMP: Learning to Physically Interact with Environments using Part-wise Motion Priorscited by 38
  37. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 37
  38. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37

50 most cited of 144 All 144 (JSON)