mlsanityData to 7 Oct 2026

IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions

arXiv 2212.07555 ↗First posted 14 Dec 2022cs.CV cs.GR cs.LGcited by 121

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full-body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated intentions of the virtual characters as input and outputs diverse sequences of full-body motions. This contrasts existing works, where full-body action synthesis methods generally do not consider object interactions, and human-object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent-driven fullbody motion generator, which uses a pair of decoupled conditional variational auto-regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6-DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state-of-the-art for the task of intent-driven motion synthesis.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 56 of its 81 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

56 of 81 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  8. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  9. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  10. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  11. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  12. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
38 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  3. 2017On human motion prediction using recurrent neural networkscited by 336
  4. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  5. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  6. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  7. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  8. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  9. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  10. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  11. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  12. 2021Character Controllers Using Motion VAEscited by 182
  13. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  14. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  15. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  16. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  17. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  18. 2018Understanding Batch Normalizationcited by 157
  19. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  20. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  21. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  22. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  23. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  24. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  25. 2018QuaterNet: A Quaternion-based Recurrent Model for Human Motioncited by 108
  26. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  27. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  28. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  29. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  30. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  31. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  32. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  33. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  34. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  35. 2021Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learningcited by 58
  36. 2021D3D-HOI: Dynamic 3D Human-Object Interactions from Videoscited by 39
  37. 2021Hand Image Understanding via Deep Multi-Task Learningcited by 31
  38. 2021Gravity-Aware Monocular 3D Human-Object Reconstructioncited by 30

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 42; 2025: 34; 2026: 28.

121 papers.

Sort:

By area:

  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  2. 2023Object Motion Guided Human Motion Synthesiscited by 206
  3. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  4. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  5. 2023Controllable Human-Object Interaction Synthesiscited by 107
  6. 2023Human Motion Generation: A Surveycited by 95
  7. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  8. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  9. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 84
  10. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  11. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  12. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
38 more
  1. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  2. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  3. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  4. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  5. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  6. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  7. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  8. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  9. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  10. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  11. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  12. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  13. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  14. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearancescited by 42
  15. 2023Task-Oriented Human-Object Interactions Generation with Implicit Neural Representationscited by 41
  16. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  17. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  18. 2023PMP: Learning to Physically Interact with Environments using Part-wise Motion Priorscited by 38
  19. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  20. 2024GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusioncited by 33
  21. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 32
  22. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  23. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  24. 2025SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Predictioncited by 26
  25. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 25
  26. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  27. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  28. 2025A Survey on Human Interaction Motion Generationcited by 25
  29. 2023ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptorscited by 23
  30. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  31. 2024ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusioncited by 21
  32. 2024GEARS: Local Geometry-aware Hand-object Interaction Synthesiscited by 20
  33. 2024Programmable Motion Generation for Open-Set Motion Control Taskscited by 20
  34. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 19
  35. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  36. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 16
  37. 2024Contact-aware Human Motion Generation from Textual Descriptionscited by 15
  38. 2024Survey on Modeling of Human-made Articulated Objectscited by 14

50 most cited of 121 All 121 (JSON)