# Physically Consistent Humanoid Loco-Manipulation using Latent Diffusion Models

arXiv: https://arxiv.org/abs/2504.16843 · HTML: https://mlsanity.com/paper/2504.16843/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 17 of 17)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence](https://mlsanity.com/paper/2311.17034/) (2023)
- [Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation](https://mlsanity.com/paper/2309.01952/) (2023)
- [Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation](https://mlsanity.com/paper/2409.20514/) (2024)
- [Versatile Multi-Contact Planning and Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2308.09179/) (2023)
- [Learning Whole-body Manipulation for Quadrupedal Robot](https://mlsanity.com/paper/2308.16820/) (2023)
- [Learning-based legged locomotion; state of the art and future perspectives](https://mlsanity.com/paper/2406.01152/) (2024)
- [Learning Force Control for Legged Manipulation](https://mlsanity.com/paper/2405.01402/) (2024)
- [RoLoMa: Robust Loco-Manipulation for Quadruped Robots with Arms](https://mlsanity.com/paper/2203.01446/) (2022)
- [Multi-contact MPC for Dynamic Loco-manipulation on Humanoid Robots](https://mlsanity.com/paper/2209.08662/) (2022)

## Abstract (arXiv)

This paper uses the capabilities of latent diffusion models (LDMs) to generate realistic RGB human-object interaction scenes to guide humanoid loco-manipulation planning. To do so, we extract from the generated images both the contact locations and robot configurations that are then used inside a whole-body trajectory optimization (TO) formulation to generate physically consistent trajectories for humanoids. We validate our full pipeline in simulation for different long-horizon loco-manipulation scenarios and perform an extensive analysis of the proposed contact and robot configuration extraction pipeline. Our results show that using the information extracted from LDMs, we can generate physically consistent trajectories that require long-horizon reasoning.