# TextIM: Part-aware Interactive Motion Synthesis from Text

arXiv: https://arxiv.org/abs/2408.03302 · HTML: https://mlsanity.com/paper/2408.03302/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping](https://mlsanity.com/paper/2112.11454/) (2021)
- [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023)
- [Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes](https://mlsanity.com/paper/2012.05522/) (2020)
- [IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions](https://mlsanity.com/paper/2212.07555/) (2022)
- [Controllable Human-Object Interaction Synthesis](https://mlsanity.com/paper/2312.03913/) (2023)
- [HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models](https://mlsanity.com/paper/2312.06553/) (2023)
- [SAGA: Stochastic Whole-Body Grasping with Contact](https://mlsanity.com/paper/2112.10103/) (2021)
- [Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis](https://mlsanity.com/paper/2205.13001/) (2022)
- [Compositional Human-Scene Interaction Synthesis with Semantic Control](https://mlsanity.com/paper/2207.12824/) (2022)

## Abstract (arXiv)

In this work, we propose TextIM, a novel framework for synthesizing TEXT-driven human Interactive Motions, with a focus on the precise alignment of part-level semantics. Existing methods often overlook the critical roles of interactive body parts and fail to adequately capture and align part-level semantics, resulting in inaccuracies and even erroneous movement outcomes. To address these issues, TextIM utilizes a decoupled conditional diffusion framework to enhance the detailed alignment between interactive movements and corresponding semantic intents from textual descriptions. Our approach leverages large language models, functioning as a human brain, to identify interacting human body parts and to comprehend interaction semantics to generate complicated and subtle interactive motion. Guided by the refined movements of the interacting parts, TextIM further extends these movements into a coherent whole-body motion. We design a spatial coherence module to complement the entire body movements while maintaining consistency and harmony across body parts using a part graph convolutional network. For training and evaluation, we carefully selected and re-labeled interactive motions from HUMANML3D to develop a specialized dataset. Experimental results demonstrate that TextIM produces semantically accurate human interactive motions, significantly enhancing the realism and applicability of synthesized interactive motions in diverse scenarios, even including interactions with deformable and dynamically changing objects.