Dynamic Motion Synthesis: Masked Audio-Text Conditioned Spatio-Temporal Transformers
Not on the map. Why?
Abstract (arXiv)
Our research presents a novel motion generation framework designed to produce whole-body motion sequences conditioned on multiple modalities simultaneously, specifically text and audio inputs. Leveraging Vector Quantized Variational Autoencoders (VQVAEs) for motion discretization and a bidirectional Masked Language Modeling (MLM) strategy for efficient token prediction, our approach achieves improved processing efficiency and coherence in the generated motions. By integrating spatial attention mechanisms and a token critic we ensure consistency and naturalness in the generated motions. This framework expands the possibilities of motion generation, addressing the limitations of existing approaches and opening avenues for multimodal motion synthesis.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 3 Sep 2024.
Builds on
32 of 57 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2017Neural Discrete Representation Learningcited by 4,650
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2021ViViT: A Video Vision Transformercited by 1,610
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
- 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
20 more
- 2020Jukebox: A Generative Model for Musiccited by 535
- 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
- 2023Simple and Controllable Music Generationcited by 480
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
- 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
- 2022AudioGen: Textually Guided Audio Generationcited by 292
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
- 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
- 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
- 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
- 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
- 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
- 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 80
- 2022NeMF: Neural Motion Fields for Kinematic Animationcited by 66
- 2022Improved Masked Image Generation with Token-Criticcited by 63
- 2020Hierarchical Quantized Autoencoderscited by 55
- 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
- 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
- 2020Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audiocited by 33
Built on by
Citing papers per year: 2025: 1.
1 paper.