# Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training

arXiv: https://arxiv.org/abs/2210.15929 · HTML: https://mlsanity.com/paper/2210.15929/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)

## Built on by (most cited first, 30 of 45)

- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [HumanTOMATO: Text-aligned Whole-body Motion Generation](https://mlsanity.com/paper/2310.12978/) (2023)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2403.01849/) (2024)
- [OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers](https://mlsanity.com/paper/2312.08985/) (2023)
- [MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling](https://mlsanity.com/paper/2409.17686/) (2024)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents](https://mlsanity.com/paper/2403.12835/) (2024)
- [Parameter-efficient Tuning of Large-scale Multimodal Foundation Model](https://mlsanity.com/paper/2305.08381/) (2023)
- [Programmable Motion Generation for Open-Set Motion Control Tasks](https://mlsanity.com/paper/2405.19283/) (2024)
- [DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding](https://mlsanity.com/paper/2411.19527/) (2024)
- [UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing](https://mlsanity.com/paper/2411.16781/) (2024)
- [Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion](https://mlsanity.com/paper/2404.11375/) (2024)
- [Learn2Talk: 3D Talking Face Learns from 2D Talking Face](https://mlsanity.com/paper/2404.12888/) (2024)
- [SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization](https://mlsanity.com/paper/2412.05095/) (2024)
- [Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases](https://mlsanity.com/paper/2310.04189/) (2023)
- [A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis](https://mlsanity.com/paper/2311.16471/) (2023)
- [Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detection](https://mlsanity.com/paper/2408.05974/) (2024)
- [SceneMI: Motion In-betweening for Modeling Human-Scene Interactions](https://mlsanity.com/paper/2503.16289/) (2025)
- [Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generations](https://mlsanity.com/paper/2412.13111/) (2024)
- [Motion Synthesis with Sparse and Flexible Keyjoint Control](https://mlsanity.com/paper/2503.15557/) (2025)
- [Textual Decomposition Then Sub-motion-space Scattering for Open-Vocabulary Motion Generation](https://mlsanity.com/paper/2411.04079/) (2024)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [Towards Robust and Controllable Text-to-Motion via Masked Autoregressive Diffusion](https://mlsanity.com/paper/2505.11013/) (2025)
- [MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion](https://mlsanity.com/paper/2403.10568/) (2024)
- [Lagrangian Motion Fields for Long-term Motion Generation](https://mlsanity.com/paper/2409.01522/) (2024)
- [DanceChat: Large Language Model-Guided Music-to-Dance Generation](https://mlsanity.com/paper/2506.10574/) (2025)
- [Temporal Consistency-Aware Text-to-Motion Generation](https://mlsanity.com/paper/2602.18057/) (2026)
- [FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions](https://mlsanity.com/paper/2312.02772/) (2023)

## Abstract (arXiv)

Text-to-motion generation is an emerging and challenging problem, which aims to synthesize motion with the same semantics as the input text. However, due to the lack of diverse labeled training data, most approaches either limit to specific types of text annotations or require online optimizations to cater to the texts during inference at the cost of efficiency and stability. In this paper, we investigate offline open-vocabulary text-to-motion generation in a zero-shot learning manner that neither requires paired training data nor extra online optimization to adapt for unseen texts. Inspired by the prompt learning in NLP, we pretrain a motion generator that learns to reconstruct the full motion from the masked motion. During inference, instead of changing the motion generator, our method reformulates the input text into a masked motion as the prompt for the motion generator to ``reconstruct'' the motion. In constructing the prompt, the unmasked poses of the prompt are synthesized by a text-to-pose generator. To supervise the optimization of the text-to-pose generator, we propose the first text-pose alignment model for measuring the alignment between texts and 3D poses. And to prevent the pose generator from overfitting to limited training texts, we further propose a novel wordless training mechanism that optimizes the text-to-pose generator without any training texts. The comprehensive experimental results show that our method obtains a significant improvement against the baseline methods. The code is available at https://github.com/junfanlin/oohmg.