mlsanityData to 7 Oct 2026

Human-Object Interaction from Human-Level Instructions

arXiv 2406.17840 ↗First posted 25 Jun 2024cs.AI cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

Intelligent agents must autonomously interact with the environments to perform daily tasks based on human-level instructions. They need a foundational understanding of the world to accurately interpret these instructions, along with precise low-level movement and interaction skills to execute the derived actions. In this work, we propose the first complete system for synthesizing physically plausible, long-horizon human-object interactions for object manipulation in contextual environments, driven by human-level instructions. We leverage large language models (LLMs) to interpret the input instructions into detailed execution plans. Unlike prior work, our system is capable of generating detailed finger-object interactions, in seamless coordination with full-body movements. We also train a policy to track generated motions in physics simulation via reinforcement learning (RL) to ensure physical plausibility of the motion. Our experiments demonstrate the effectiveness of our system in synthesizing realistic interactions with diverse objects in complex environments, highlighting its potential for real-world applications.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

54 of 79 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Video Diffusion Modelscited by 2,280
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  6. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  7. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  8. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  9. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  10. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  11. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  12. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
38 more
  1. 2023Object Motion Guided Human Motion Synthesiscited by 206
  2. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  3. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  4. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  5. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  6. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  7. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  8. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  9. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  10. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  11. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  12. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  13. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  14. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  15. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  16. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 129
  17. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  18. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  19. 2023Controllable Human-Object Interaction Synthesiscited by 107
  20. 2023Synthesizing Physical Character-Scene Interactionscited by 102
  21. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 101
  22. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  23. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  24. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  25. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  26. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  27. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  28. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  29. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  30. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  31. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  32. 2021Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimatorcited by 70
  33. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  34. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  35. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  36. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  37. 2022FLEX: Full-Body Grasping Without Full-Body Graspscited by 46
  38. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 28; 2026: 31.

62 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  2. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 57
  3. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  4. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  5. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  6. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  7. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  8. 2025Learning Getting-Up Policies for Real-World Humanoid Robotscited by 21
  9. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  10. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 16
  11. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  12. 2025PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learningcited by 13
38 more
  1. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  2. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  3. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 10
  4. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 10
  5. 2025HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusioncited by 10
  6. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  7. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  8. 2026FrankenMotion: Part-level Human Motion Generation and Compositioncited by 7
  9. 20253D Human Interaction Generation: A Surveycited by 6
  10. 2025HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perceptioncited by 5
  11. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  12. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Modelscited by 4
  13. 2025RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrationscited by 4
  14. 2025Half-Physics: Enabling Kinematic 3D Human Model with Physical Interactionscited by 4
  15. 2025Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contextscited by 4
  16. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  17. 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Sizecited by 4
  18. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 4
  19. 2026SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interactioncited by 4
  20. 2025SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learningcited by 3
  21. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangementcited by 3
  22. 2026Disentangled Hierarchical VAE for 3D Human-Human Interaction Generationcited by 3
  23. 2026InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinementcited by 3
  24. 2025Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motionscited by 2
  25. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 2
  26. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 2
  27. 2026VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenescited by 2
  28. 2025InterPose: Learning to Generate Human-Object Interactions from Large-Scale Web Videoscited by 1
  29. 2026DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitationcited by 1
  30. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 1
  31. 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scenecited by 1
  32. 2026Dex2HOI: Dexterous Bimanual Two-Object Interaction Generationcited by 1
  33. 2026MOCHI: Motion Enhancement of Collaborative Human-object Interactionscited by 1
  34. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  35. 2026HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generationcited by 1
  36. 2025Generating Detailed Character Motion from Blocking Posescited by 0
  37. 2025BRIC: Bridging Kinematic Plans and Physical Control at Test Timecited by 0
  38. 2026SIAgent: Spatial Interaction Agent via LLM-powered Eye-Hand Motion Intent Understanding in VRcited by 0

50 most cited of 62 All 62 (JSON)