# You Think, You ACT: The New Task of Arbitrary Text to Motion Generation arXiv: https://arxiv.org/abs/2404.14745 ยท HTML: https://mlsanity.com/paper/2404.14745/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 45 of its 64 references that are in the map. - First posted on arXiv 23 Apr 2024. ## Builds on (most cited first, 30 of 45) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019) - [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022) - [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022) - [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016) - [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023) - [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022) - [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019) - [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021) - [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022) - [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021) - [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020) - [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022) - [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023) - [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022) - [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016) - [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021) - [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023) - [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021) - [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019) - [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021) - [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023) - [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023) - [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023) - [MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators](https://mlsanity.com/paper/2306.10900/) (2023) - [CG-HOI: Contact-Guided 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2311.16097/) (2023) - [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024) - [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023) - [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021) ## Built on by (most cited first, 8 of 8) - [NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering](https://mlsanity.com/paper/2503.07076/) (2025) - [LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens](https://mlsanity.com/paper/2602.12370/) (2026) - [Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learning](https://mlsanity.com/paper/2605.30969/) (2026) - [MotionHiFlow: Text-to-motion via hierarchical flow matching](https://mlsanity.com/paper/2604.23264/) (2026) - [SimROD: A Simple Baseline for Raw Object Detection with Global and Local Enhancements](https://mlsanity.com/paper/2503.07101/) (2025) - [Spatial Temporal Synergy: Balancing Change and Invariance in Text Driven 3D Human Motion Editing](https://mlsanity.com/paper/2608.16008/) (2026) - [Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control](https://mlsanity.com/paper/2605.14935/) (2026) - [MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations](https://mlsanity.com/paper/2606.27779/) (2026) ## Abstract (arXiv) Text to Motion aims to generate human motions from texts. Existing settings rely on limited Action Texts that include action labels, which limits flexibility and practicability in scenarios difficult to describe directly. This paper extends limited Action Texts to arbitrary ones. Scene texts without explicit action labels can enhance the practicality of models in complex and diverse industries such as virtual human interaction, robot behavior generation, and film production, while also supporting the exploration of potential implicit behavior patterns. However, newly introduced Scene Texts may yield multiple reasonable output results, causing significant challenges in existing data, framework, and evaluation. To address this practical issue, we first create a new dataset HUMANML3D++ by extending texts of the largest existing dataset HUMANML3D. Secondly, we propose a simple yet effective framework that extracts action instructions from arbitrary texts and subsequently generates motions. Furthermore, we also benchmark this new setting with multi-solution metrics to address the inadequacies of existing single-solution metrics. Extensive experiments indicate that Text to Motion in this realistic setting is challenging, fostering new research in this practical direction.