mlsanityData to 14 Oct 2026

HumanTOMATO: Text-aligned Whole-body Motion Generation

arXiv 2310.12978 ↗First posted 19 Oct 2023cs.CVcited by 104

Not on the map. Why?

Abstract (arXiv)

This work targets a novel text-driven whole-body motion generation task, which takes a given textual description as input and aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously. Previous works on text-driven motion generation tasks mainly have two limitations: they ignore the key role of fine-grained hand and face controlling in vivid whole-body motion generation, and lack a good alignment between text and motion. To address such limitations, we propose a Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which is the first attempt to our knowledge towards applicable holistic motion generation in this research area. To tackle this challenging task, our solution includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H2VQ) and a Hierarchical-GPT for fine-grained body and hand motion reconstruction and generation with two structured codebooks; and (2) a pre-trained text-motion-alignment model to help generated motion align with the input textual description explicitly. Comprehensive experiments verify that our model has significant advantages in both the quality of generated motions and their alignment with text.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 66 of its 80 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

66 of 80 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  11. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  12. 2022Human Motion Diffusion Modelcited by 1,080
38 more
  1. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  6. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 583
  7. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  8. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  9. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  10. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  11. 2023Simple and Controllable Music Generationcited by 488
  12. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  13. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  14. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  15. 2016The KIT Motion-Language Datasetcited by 338
  16. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 307
  17. 2022EDGE: Editable Dance Generation From Musiccited by 287
  18. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 265
  19. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  20. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  21. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  22. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  23. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  24. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  25. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
  26. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  27. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  28. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  29. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  30. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  31. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  32. 2020Skeleton-Aware Networks for Deep Motion Retargetingcited by 124
  33. 2022Selective-Supervised Contrastive Learning with Noisy Labelscited by 112
  34. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  35. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  36. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  37. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  38. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 88

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 37; 2025: 43; 2026: 20.

104 papers.

Sort:

By area:

  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  2. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 168
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  4. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  5. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  6. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 86
  7. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  8. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  9. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  10. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  11. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  12. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 59
38 more
  1. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  2. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
  3. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  4. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 45
  5. 2024MoGenTS: Motion Generation based on Spatial-Temporal Joint Modelingcited by 41
  6. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  7. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  8. 2024MotionChain: Conversational Motion Controllers via Multimodal Promptscited by 30
  9. 2025MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigmcited by 30
  10. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 26
  11. 2024Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learningcited by 25
  12. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 22
  13. 2024Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patternscited by 21
  14. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 21
  15. 2025Referring to Any Personcited by 21
  16. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 20
  17. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  18. 2025Motion2Motion: Cross-topology Motion Transfer with Sparse Correspondencecited by 19
  19. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 18
  20. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  21. 2024Contact-aware Human Motion Generation from Textual Descriptionscited by 16
  22. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  23. 2025HERO: Human Reaction Generation from Videoscited by 15
  24. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  25. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  26. 2024Diffgrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Modelcited by 13
  27. 2025DPoser-X: Diffusion Model as Robust 3D Whole-body Human Pose Priorcited by 13
  28. 2023A Unified Framework for Multimodal, Multi-Part Human Motion Synthesiscited by 12
  29. 2025Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Controlcited by 12
  30. 2025MOSPA: Human Motion Generation Driven by Spatial Audiocited by 12
  31. 2024Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Spacecited by 10
  32. 2024MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learningcited by 10
  33. 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Modelcited by 10
  34. 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructionscited by 10
  35. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  36. 2025MotionDreamer: One-to-Many Motion Synthesis with Localized Generative Masked Transformercited by 9
  37. 2024You Think, You ACT: The New Task of Arbitrary Text to Motion Generationcited by 8
  38. 2024MotionWavelet: Human Motion Prediction via Wavelet Manifold Learningcited by 8

50 most cited of 104 All 104 (JSON)