mlsanityData to 14 Oct 2026

Text2Action: Generative Adversarial Synthesis from Language to Action

arXiv 1710.05298 ↗First posted 15 Oct 2017cs.LG cs.CL cs.ROcited by 106

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose a generative model which learns the relationship between language and human action in order to generate a human action sequence given a sentence describing human behavior. The proposed generative model is a generative adversarial network (GAN), which is based on the sequence to sequence (SEQ2SEQ) model. Using the proposed generative network, we can synthesize various actions for a robot or a virtual agent using a text encoder recurrent neural network (RNN) and an action decoder RNN. The proposed generative network is trained from 29,770 pairs of actions and sentence annotations extracted from MSR-Video-to-Text (MSR-VTT), a large-scale video dataset. We demonstrate that the network can generate human-like actions which can be transferred to a Baxter robot, such that the robot performs an action based on a provided sentence. Results show that the proposed generative network correctly models the relationship between language and action and can generate a diverse set of actions from the same sentence.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 16; 2023: 22; 2024: 32; 2025: 19; 2026: 11.

106 papers.

Sort:

By area:

  1. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  2. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  3. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  4. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  5. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  6. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  7. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 206
  8. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  9. 2022TEACH: Temporal Action Composition for 3D Humanscited by 176
  10. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 168
  11. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 124
  12. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 117
38 more
  1. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 104
  2. 2023Human Motion Generation: A Surveycited by 96
  3. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  4. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  5. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  6. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  7. 2023Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Modelcited by 79
  8. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  9. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
  10. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  11. 2024Flexible Motion In-betweening with Diffusion Modelscited by 71
  12. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  13. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 64
  14. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 60
  15. 2022AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatarscited by 55
  16. 2024ParCo: Part-Coordinating Text-to-Motion Synthesiscited by 55
  17. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
  18. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  19. 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
  20. 2022MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labelscited by 45
  21. 2023BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesiscited by 45
  22. 2023Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphscited by 45
  23. 2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearancescited by 42
  24. 2022CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshescited by 41
  25. 2023PoseFix: Correcting 3D Human Poses with Natural Languagecited by 40
  26. 2023Plan, Posture and Go: Towards Open-World Text-to-Motion Generationcited by 26
  27. 2022Conditional Motion In-betweeningcited by 25
  28. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  29. 2024M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Modelscited by 24
  30. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 22
  31. 2023Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videoscited by 21
  32. 2024HUMOS: Human Motion Model Conditioned on Body Shapecited by 20
  33. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 18
  34. 2024A Cross-Dataset Study for Text-based 3D Human Motion Retrievalcited by 17
  35. 2021Action2video: Generating Videos of Human 3D Actionscited by 15
  36. 2021MUGL: Large Scale Multi Person Conditional Action Generation with Locomotioncited by 14
  37. 2022PoseScript: Linking 3D Human Poses and Natural Languagecited by 14
  38. 2024T2LM: Long-Term 3D Human Motion Generation from Multiple Sentencescited by 12

50 most cited of 106 All 106 (JSON)