# Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis

arXiv: https://arxiv.org/abs/2503.00371 · HTML: https://mlsanity.com/paper/2503.00371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 98 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes](https://mlsanity.com/paper/2210.09729/) (2022)
- [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024)

## Built on by (most cited first, 2 of 2)

- [NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering](https://mlsanity.com/paper/2603.20187/) (2026)
- [DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation](https://mlsanity.com/paper/2606.14721/) (2026)

## Abstract (arXiv)

As two intimate reciprocal tasks, scene-aware human motion synthesis and analysis require a joint understanding between multiple modalities, including 3D body motions, 3D scenes, and textual descriptions. In this paper, we integrate these two paired processes into a Co-Evolving Synthesis-Analysis (CESA) pipeline and mutually benefit their learning. Specifically, scene-aware text-to-human synthesis generates diverse indoor motion samples from the same textual description to enrich human-scene interaction intra-class diversity, thus significantly benefiting training a robust human motion analysis system. Reciprocally, human motion analysis would enforce semantic scrutiny on each synthesized motion sample to ensure its semantic consistency with the given textual description, thus improving realistic motion synthesis. Considering that real-world indoor human motions are goal-oriented and path-guided, we propose a cascaded generation strategy that factorizes text-driven scene-specific human motion generation into three stages: goal inferring, path planning, and pose synthesizing. Coupling CESA with this powerful cascaded motion synthesis model, we jointly improve realistic human motion synthesis and robust human motion analysis in 3D scenes.