mlsanityData to 7 Oct 2026

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

arXiv 2104.05670 ↗First posted 12 Apr 2021cs.CVcited by 497

Not on the map. Why?

Cited by 99 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle the problem of action-conditioned generation of realistic and diverse human motion sequences. In contrast to methods that complete, or extend, motion sequences, this task does not require an initial pose or sequence. Here we learn an action-aware latent representation for human motions by training a generative variational autoencoder (VAE). By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action. Specifically, we design a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets. We evaluate our approach on the NTU RGB+D, HumanAct12 and UESTC datasets and show improvements over the state of the art. Furthermore, we present two use cases: improving action recognition through adding our synthesized data to training, and motion denoising. Code and models are available on our project page.

Key facts. Cited by 497 papers in the map (data through 7 Oct 2026). Builds on 42 of its 83 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

42 of 83 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  11. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  12. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
30 more
  1. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  2. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  3. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  4. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  5. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  6. 2017On human motion prediction using recurrent neural networkscited by 336
  7. 2018Learning 3D Human Dynamics from Videocited by 315
  8. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  9. 2021Robust Motion In-betweeningcited by 223
  10. 2020STAR: Sparse Trained Articulated Human Body Regressorcited by 194
  11. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  12. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  13. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  14. 2018QuaterNet: A Quaternion-based Recurrent Model for Human Motioncited by 108
  15. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  16. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  17. 2019Structured Prediction Helps 3D Human Motion Modellingcited by 96
  18. 2020Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flowscited by 94
  19. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  20. 2019Context-aware Human Motion Predictioncited by 83
  21. 2021Single-Shot Motion Completion with Transformercited by 59
  22. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 50
  23. 2019Synthetic Humans for Action Recognition from Unseen Viewpointscited by 46
  24. 2018Human Motion Modeling using DVGANscited by 45
  25. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  26. 2020Variational Transformers for Diverse Response Generationcited by 36
  27. 2020Perpetual Motion: Generating Unbounded Human Motioncited by 36
  28. 2019Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modelingcited by 32
  29. 20203D Human Shape Reconstruction from a Polarization Imagecited by 28
  30. 2018Variational Semi-supervised Aspect-term Sentiment Analysis via Transformercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 66; 2023: 122; 2024: 123; 2025: 109; 2026: 67.

497 papers.

Sort:

By area:

  1. 2022Human Motion Diffusion Modelcited by 1,062
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  3. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  4. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  5. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  6. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  7. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  8. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  9. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  10. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  11. 2022EDGE: Editable Dance Generation From Musiccited by 284
  12. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
38 more
  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  2. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  3. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  4. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  5. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  7. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  8. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  9. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  10. 2022TEACH: Temporal Action Composition for 3D Humanscited by 173
  11. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  12. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  13. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  14. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  15. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  16. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  17. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  18. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  19. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  20. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 116
  21. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  22. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  23. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  24. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  25. 2023Controllable Human-Object Interaction Synthesiscited by 107
  26. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 105
  27. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 104
  28. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  29. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  30. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  31. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  32. 2023Human Motion Generation: A Surveycited by 95
  33. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  34. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  35. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  36. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  37. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  38. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82

50 most cited of 497 All 497 (JSON)