# Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases arXiv: https://arxiv.org/abs/2310.04189 ยท HTML: https://mlsanity.com/paper/2310.04189/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 47 of its 75 references that are in the map. - First posted on arXiv 6 Oct 2023. ## Builds on (most cited first, 30 of 47) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019) - [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019) - [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018) - [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022) - [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022) - [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023) - [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022) - [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023) - [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021) - [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022) - [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021) - [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020) - [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022) - [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020) - [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022) - [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022) - [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016) - [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023) - [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021) - [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016) - [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022) - [Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset](https://mlsanity.com/paper/2307.00818/) (2023) - [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023) - [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022) - [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021) - [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022) - [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023) - [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022) - [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022) ## Built on by (most cited first, 13 of 13) - [Light-T2M: A Lightweight and Fast Model for Text-to-motion Generation](https://mlsanity.com/paper/2412.11193/) (2024) - [KinMo: Kinematic-aware Human Motion Understanding and Generation](https://mlsanity.com/paper/2411.15472/) (2024) - [ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generation](https://mlsanity.com/paper/2503.13130/) (2025) - [MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations](https://mlsanity.com/paper/2410.13790/) (2024) - [Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions](https://mlsanity.com/paper/2503.15898/) (2025) - [ImDy: Human Inverse Dynamics from Imitated Observations](https://mlsanity.com/paper/2410.17610/) (2024) - [Progressive Human Motion Generation Based on Text and Few Motion Frames](https://mlsanity.com/paper/2503.13300/) (2025) - [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024) - [MotionHiFlow: Text-to-motion via hierarchical flow matching](https://mlsanity.com/paper/2604.23264/) (2026) - [HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generation](https://mlsanity.com/paper/2608.15141/) (2026) - [SOSControl: Enhancing Human Motion Generation through Saliency-Aware Symbolic Orientation and Timing Control](https://mlsanity.com/paper/2601.14258/) (2025) - [T2MBench: A Benchmark for Out-of-Distribution Text-to-Motion Generation](https://mlsanity.com/paper/2602.13751/) (2026) - [LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference](https://mlsanity.com/paper/2603.11605/) (2026) ## Abstract (arXiv) Motion understanding aims to establish a reliable mapping between motion and action semantics, while it is a challenging many-to-many problem. An abstract action semantic (i.e., walk forwards) could be conveyed by perceptually diverse motions (walking with arms up or swinging). In contrast, a motion could carry different semantics w.r.t. its context and intention. This makes an elegant mapping between them difficult. Previous attempts adopted direct-mapping paradigms with limited reliability. Also, current automatic metrics fail to provide reliable assessments of the consistency between motions and action semantics. We identify the source of these problems as the significant gap between the two modalities. To alleviate this gap, we propose Kinematic Phrases (KP) that take the objective kinematic facts of human motion with proper abstraction, interpretability, and generality. Based on KP, we can unify a motion knowledge base and build a motion understanding system. Meanwhile, KP can be automatically converted from motions to text descriptions with no subjective bias, inspiring Kinematic Prompt Generation (KPG) as a novel white-box motion generation benchmark. In extensive experiments, our approach shows superiority over other methods. Our project is available at https://foruck.github.io/KP/.