mlsanityData to 14 Oct 2026

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

arXiv 2207.01696 ↗First posted 4 Jul 2022cs.CVcited by 359

Not on the map. Why?

Cited by 70 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Inspired by the strong ties between vision and language, the two intimate human sensing and communication modalities, our paper aims to explore the generation of 3D human full-body motions from texts, as well as its reciprocal task, shorthanded for text2motion and motion2text, respectively. To tackle the existing challenges, especially to enable the generation of multiple distinct motions from the same text, and to avoid the undesirable production of trivial motionless pose sequences, we propose the use of motion token, a discrete and compact motion representation. This provides one level playing ground when considering both motions and text signals, as the motion and text tokens, respectively. Moreover, our motion2text module is integrated into the inverse alignment process of our text2motion training pipeline, where a significant deviation of synthesized text from the input text would be penalized by a large training loss; empirically this is shown to effectively improve performance. Finally, the mappings in-between the two modalities of motions and texts are facilitated by adapting the neural model for machine translation (NMT) to our context. This autoregressive modeling of the distribution over discrete motion tokens further enables non-deterministic production of pose sequences, of variable lengths, from an input text. Our approach is flexible, could be used for both text2motion and motion2text tasks. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach on both tasks over a variety of state-of-the-art methods. Project page: https://ericguo5513.github.io/TM2T/

Key facts. Cited by 359 papers in the map (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 4 Jul 2022.

Builds on

43 of 63 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  11. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
31 more
  1. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  2. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  3. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  4. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  5. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  6. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
  7. 2016The KIT Motion-Language Datasetcited by 338
  8. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  9. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  10. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 185
  11. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  12. 2015Sequence to Sequence -- Video to Textcited by 158
  13. 2017Video Generation From Textcited by 154
  14. 2020Latent Video Transformercited by 106
  15. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  16. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  17. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  18. 2019Context-aware Human Motion Predictioncited by 84
  19. 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
  20. 2021Generating Smooth Pose Sequences for Diverse Human Motion Predictioncited by 66
  21. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  22. 2017Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networkscited by 61
  23. 2020Socially and Contextually Aware Human Motion and Pose Forecastingcited by 51
  24. 2019Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitionscited by 46
  25. 2019Modeling Human Motion with Quaternion-based Neural Networkscited by 45
  26. 2018Adversarial Inference for Multi-Sentence Video Descriptioncited by 31
  27. 2020Structure-Aware Human-Action Generationcited by 28
  28. 2021Action2video: Generating Videos of Human 3D Actionscited by 15
  29. 2019Self-critical n-step Training for Image Captioningcited by 12
  30. 2016Lie-X: Depth Image Based Articulated Object Pose Estimation, Tracking, and Action Recognition on Lie Groupscited by 10
  31. 2021Label-Attention Transformer with Geometrically Coherent Objects for Image Captioningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 54; 2024: 98; 2025: 119; 2026: 82.

359 papers.

Sort:

By area:

  1. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  2. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  3. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  4. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  5. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 287
  6. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
  7. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  8. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  9. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 206
  10. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  11. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 168
  12. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
38 more
  1. 2023MMM: Generative Masked Motion Modelcited by 131
  2. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 117
  3. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  4. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 114
  5. 2023Controllable Human-Object Interaction Synthesiscited by 109
  6. 2023FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editingcited by 106
  7. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  8. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 98
  9. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  10. 2023Human Motion Generation: A Surveycited by 96
  11. 2024TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representationcited by 95
  12. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 94
  13. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  14. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  15. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 89
  16. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 86
  17. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 86
  18. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 85
  19. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 80
  20. 2023Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Modelcited by 79
  21. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  22. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  23. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  24. 2023Priority-Centric Human Motion Generation in Discrete Latent Spacecited by 67
  25. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 67
  26. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  27. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 64
  28. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  29. 2024MotionFix: Text-Driven 3D Human Motion Editingcited by 64
  30. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
  31. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  32. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 60
  33. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 59
  34. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  35. 2024ParCo: Part-Coordinating Text-to-Motion Synthesiscited by 55
  36. 2024BAMM: Bidirectional Autoregressive Motion Modelcited by 55
  37. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
  38. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49

50 most cited of 359 All 359 (JSON)