Hierarchical Planning and Control for Box Loco-Manipulation
Not on the map. Why?
Abstract (arXiv)
Humans perform everyday tasks using a combination of locomotion and manipulation skills. Building a system that can handle both skills is essential to creating virtual humans. We present a physically-simulated human capable of solving box rearrangement tasks, which requires a combination of both skills. We propose a hierarchical control architecture, where each level solves the task at a different level of abstraction, and the result is a physics-based simulated virtual human capable of rearranging boxes in a cluttered environment. The control architecture integrates a planner, diffusion models, and physics-based motion imitation of sparse motion clips using deep reinforcement learning. Boxes can vary in size, weight, shape, and placement height. Code and trained control policies are provided.
Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Jun 2023.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2023: 6; 2024: 13; 2025: 13; 2026: 12.
44 papers.
Sort:
By area:
- 2023Object Motion Guided Human Motion Synthesiscited by 212
- 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
- 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
- 2023Controllable Human-Object Interaction Synthesiscited by 109
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
- 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
- 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
- 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
- 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
- 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
- 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
32 more
- 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 44
- 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
- 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
- 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Controlcited by 28
- 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
- 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
- 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 20
- 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction Systemcited by 18
- 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
- 2024SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generationcited by 12
- 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 12
- 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 11
- 2024F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performancecited by 10
- 2026HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulationcited by 10
- 2024FORCE: Physics-aware Human-object Interactioncited by 7
- 2025Learning to Ball: Composing Policies for Long-Horizon Basketball Movescited by 6
- 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5
- 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Sizecited by 5
- 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesiscited by 4
- 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
- 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic modelscited by 3
- 2025Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulationcited by 3
- 2026ViHOI: Human-Object Interaction Synthesis with Visual Priorscited by 3
- 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learningcited by 2
- 2026DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitationcited by 1
- 2026MUSIC: Learning Muscle-Driven Dexterous Hand Controlcited by 1
- 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scenecited by 1
- 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenescited by 0
- 2026Robo-Saber: Generating and Simulating Virtual Reality Playerscited by 0
- 2026Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robotcited by 0
- 2026MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generationcited by 0
- 2026Scalable Multi-Human Cooperative Manipulation via Commander-Executor Coordinationcited by 0
- 2026Scalable Multi-Human Cooperative Manipulation via Commander-Executor Coordination10 Oct 2026
- 2026MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation7 May 2026
- 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scene1 May 2026
- 2026MUSIC: Learning Muscle-Driven Dexterous Hand Control26 Apr 2026
- 2026Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robot23 Apr 2026
- 2026DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation22 Apr 2026
- 2026HEX: Humanoid-Aligned Experts for Cross-Embodiment Whole-Body Manipulation9 Apr 2026
- 2026ViHOI: Human-Object Interaction Synthesis with Visual Priors25 Mar 2026
- 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning14 Mar 2026
- 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size9 Mar 2026
- 2026Robo-Saber: Generating and Simulating Virtual Reality Players20 Feb 2026
- 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions5 Feb 2026
32 more
- 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System13 Oct 2025
- 2025Learning to Ball: Composing Policies for Long-Horizon Basketball Moves26 Sep 2025
- 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation11 Sep 2025
- 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes23 Aug 2025
- 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning18 May 2025
- 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis14 Apr 2025
- 2025Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation13 Apr 2025
- 2025Guiding Human-Object Interactions with Rich Geometry and Relations26 Mar 2025
- 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization25 Mar 2025
- 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy24 Mar 2025
- 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation17 Mar 2025
- 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control14 Mar 2025
- 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions27 Feb 2025
- 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation24 Dec 2024
- 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking23 Dec 2024
- 2024SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation29 Nov 2024
- 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic models25 Nov 2024
- 2024F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance8 Oct 2024
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control4 Oct 2024
- 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models30 Sep 2024
- 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement27 Jun 2024
- 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics20 Jun 2024
- 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction28 Mar 2024
- 2024FORCE: Physics-aware Human-object Interaction17 Mar 2024
- 2024THOR: Text to Human-Object Interaction Diffusion via Relation Intervention17 Mar 2024
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation15 Mar 2024
- 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction7 Dec 2023
- 2023Controllable Human-Object Interaction Synthesis6 Dec 2023
- 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generation27 Nov 2023
- 2023Object Motion Guided Human Motion Synthesis28 Sep 2023
- 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion31 Aug 2023
- 2023Synthesizing Physically Plausible Human Motions in 3D Scenes17 Aug 2023
- 2023Controllable Human-Object Interaction Synthesiscited by 109
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
- 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
- 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
- 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Controlcited by 28
- 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesiscited by 4
- 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
- 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic modelscited by 3
- 2026ViHOI: Human-Object Interaction Synthesis with Visual Priorscited by 3
- 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learningcited by 2
- 2026ViHOI: Human-Object Interaction Synthesis with Visual Priors25 Mar 2026
- 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning14 Mar 2026
- 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis14 Apr 2025
- 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control14 Mar 2025
- 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic models25 Nov 2024
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control4 Oct 2024
- 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models30 Sep 2024
- 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction28 Mar 2024
- 2024THOR: Text to Human-Object Interaction Diffusion via Relation Intervention17 Mar 2024
- 2023Controllable Human-Object Interaction Synthesis6 Dec 2023
- 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 11
- 2025Learning to Ball: Composing Policies for Long-Horizon Basketball Movescited by 6
- 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learningcited by 5
- 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenescited by 0
- 2026Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robotcited by 0
- 2026Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robot23 Apr 2026
- 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions5 Feb 2026
- 2025Learning to Ball: Composing Policies for Long-Horizon Basketball Moves26 Sep 2025
- 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes23 Aug 2025
- 2025Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning18 May 2025
- 2023Object Motion Guided Human Motion Synthesiscited by 212
- 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
- 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 138
- 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
- 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
- 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
- 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
- 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
- 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 44
- 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
- 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
- 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
12 most cited of 26 All 26 (JSON)
- 2026Scalable Multi-Human Cooperative Manipulation via Commander-Executor Coordination10 Oct 2026
- 2026MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation7 May 2026
- 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scene1 May 2026
- 2026MUSIC: Learning Muscle-Driven Dexterous Hand Control26 Apr 2026
- 2026DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation22 Apr 2026
- 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size9 Mar 2026
- 2026Robo-Saber: Generating and Simulating Virtual Reality Players20 Feb 2026
- 2025PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System13 Oct 2025
- 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation11 Sep 2025
- 2025Guiding Human-Object Interactions with Rich Geometry and Relations26 Mar 2025
- 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization25 Mar 2025
- 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy24 Mar 2025
12 newest of 26 All 26 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation
- 2024HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation
- 2026MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulation
- 2026FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences
- 2026LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
- 2025SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending
- 2026KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation ↗
- 2026EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation