# PMP: Learning to Physically Interact with Environments using Part-wise Motion Priors

arXiv: https://arxiv.org/abs/2305.03249 · HTML: https://mlsanity.com/paper/2305.03249/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions](https://mlsanity.com/paper/2212.07555/) (2022)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [SAGA: Stochastic Whole-Body Grasping with Contact](https://mlsanity.com/paper/2112.10103/) (2021)
- [Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis](https://mlsanity.com/paper/2205.13001/) (2022)
- [Motion Puzzle: Arbitrary Motion Style Transfer by Body Part](https://mlsanity.com/paper/2202.05274/) (2022)
- [FLEX: Full-Body Grasping Without Full-Body Grasps](https://mlsanity.com/paper/2211.11903/) (2022)
- [Real-time Controllable Motion Transition for Characters](https://mlsanity.com/paper/2205.02540/) (2022)
- [Learning to Use Chopsticks in Diverse Gripping Styles](https://mlsanity.com/paper/2205.14313/) (2022)
- [ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters](https://mlsanity.com/paper/2205.01906/) (2022)

## Built on by (most cited first, 30 of 38)

- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [TLControl: Trajectory and Language Control for Human Motion Synthesis](https://mlsanity.com/paper/2311.17135/) (2023)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction](https://mlsanity.com/paper/2403.19652/) (2024)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [THOR: Text to Human-Object Interaction Diffusion via Relation Intervention](https://mlsanity.com/paper/2403.11208/) (2024)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [SMPLOlympics: Sports Environments for Physically Simulated Humanoids](https://mlsanity.com/paper/2407.00187/) (2024)
- [SkillMimic-V2: Learning Robust and Generalizable Interaction Skills from Sparse and Noisy Demonstrations](https://mlsanity.com/paper/2505.02094/) (2025)
- [PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios](https://mlsanity.com/paper/2404.19722/) (2024)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [LGTM: Local-to-Global Text-Driven Human Motion Diffusion Model](https://mlsanity.com/paper/2405.03485/) (2024)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [SceneMI: Motion In-betweening for Modeling Human-Scene Interactions](https://mlsanity.com/paper/2503.16289/) (2025)
- [InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2602.06035/) (2026)
- [Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation](https://mlsanity.com/paper/2410.13817/) (2024)
- [Motion Synthesis with Sparse and Flexible Keyjoint Control](https://mlsanity.com/paper/2503.15557/) (2025)
- [ModSkill: Physical Character Skill Modularization](https://mlsanity.com/paper/2502.14140/) (2025)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs](https://mlsanity.com/paper/2502.05641/) (2025)
- [Learning to Ball: Composing Policies for Long-Horizon Basketball Moves](https://mlsanity.com/paper/2509.22442/) (2025)
- [Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning](https://mlsanity.com/paper/2505.12278/) (2025)
- [TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size](https://mlsanity.com/paper/2603.07988/) (2026)
- [CBIL: Collective Behavior Imitation Learning for Fish from Real Videos](https://mlsanity.com/paper/2504.00234/) (2025)
- [COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models](https://mlsanity.com/paper/2409.20502/) (2024)
- [ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenes](https://mlsanity.com/paper/2412.08685/) (2024)
- [Versatile Physics-based Character Control with Hybrid Latent Representation](https://mlsanity.com/paper/2503.12814/) (2025)

## Abstract (arXiv)

We present a method to animate a character incorporating multiple part-wise motion priors (PMP). While previous works allow creating realistic articulated motions from reference data, the range of motion is largely limited by the available samples. Especially for the interaction-rich scenarios, it is impractical to attempt acquiring every possible interacting motion, as the combination of physical parameters increases exponentially. The proposed PMP allows us to assemble multiple part skills to animate a character, creating a diverse set of motions with different combinations of existing data. In our pipeline, we can train an agent with a wide range of part-wise priors. Therefore, each body part can obtain a kinematic insight of the style from the motion captures, or at the same time extract dynamics-related information from the additional part-specific simulation. For example, we can first train a general interaction skill, e.g. grasping, only for the dexterous part, and then combine the expert trajectories from the pre-trained agent with the kinematic priors of other limbs. Eventually, our whole-body agent learns a novel physical interaction skill even with the absence of the object trajectories in the reference motion sequence.