# Contact-aware Human Motion Generation from Textual Descriptions

arXiv: https://arxiv.org/abs/2403.15709 · HTML: https://mlsanity.com/paper/2403.15709/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 62 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)

## Built on by (most cited first, 16 of 16)

- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction](https://mlsanity.com/paper/2403.19652/) (2024)
- [Towards Variable and Coordinated Holistic Co-Speech Motion Generation](https://mlsanity.com/paper/2404.00368/) (2024)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control](https://mlsanity.com/paper/2412.15664/) (2024)
- [Human-Aware 3D Scene Generation with Spatially-constrained Diffusion Models](https://mlsanity.com/paper/2406.18159/) (2024)
- [COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models](https://mlsanity.com/paper/2409.20502/) (2024)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)
- [Towards Consistent Long-Term Pose Generation](https://mlsanity.com/paper/2507.18382/) (2025)
- [GIRAF: Towards Generalizable Human Interactions with Articulated Objects](https://mlsanity.com/paper/2607.07880/) (2026)
- [TextIM: Part-aware Interactive Motion Synthesis from Text](https://mlsanity.com/paper/2408.03302/) (2024)
- [Talking Together: Synthesizing Co-Located 3D Conversations from Audio](https://mlsanity.com/paper/2603.08674/) (2026)
- [Contact Matrix: Enhancing Dance Motion Synthesis with Precise Interaction Modeling](https://mlsanity.com/paper/2605.04662/) (2026)
- [Multi-Modal Controlled Coherent Motion Generation](https://mlsanity.com/paper/2609.11439/) (2026)
- [PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation](https://mlsanity.com/paper/2609.38466/) (2026)

## Abstract (arXiv)

This paper addresses the problem of generating 3D interactive human motion from text. Given a textual description depicting the actions of different body parts in contact with static objects, we synthesize sequences of 3D body poses that are visually natural and physically plausible. Yet, this task poses a significant challenge due to the inadequate consideration of interactions by physical contacts in both motion and textual descriptions, leading to unnatural and implausible sequences. To tackle this challenge, we create a novel dataset named RICH-CAT, representing "Contact-Aware Texts" constructed from the RICH dataset. RICH-CAT comprises high-quality motion, accurate human-object contact labels, and detailed textual descriptions, encompassing over 8,500 motion-text pairs across 26 indoor/outdoor actions. Leveraging RICH-CAT, we propose a novel approach named CATMO for text-driven interactive human motion synthesis that explicitly integrates human body contacts as evidence. We employ two VQ-VAE models to encode motion and body contact sequences into distinct yet complementary latent spaces and an intertwined GPT for generating human motions and contacts in a mutually conditioned manner. Additionally, we introduce a pre-trained text encoder to learn textual embeddings that better discriminate among various contact types, allowing for more precise control over synthesized motions and contacts. Our experiments demonstrate the superior performance of our approach compared to existing text-to-motion methods, producing stable, contact-aware motion sequences. Code and data will be available for research purposes at https://xymsh.github.io/RICH-CAT/