mlsanityData to 14 Oct 2026

MoSAT: Human Motion Generation from Spatial Audio and Textual Description

arXiv 2609.23797 ↗First posted 20 Sep 2026cs.GR cs.CV cs.ROcited by 1

Not on the map. Why?

Abstract (arXiv)

Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 64 of its 75 references that are in the map. First posted on arXiv 20 Sep 2026.

Builds on

64 of 75 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  7. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  9. 2022Human Motion Diffusion Modelcited by 1,080
  10. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  11. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  12. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
38 more
  1. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  2. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  3. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  4. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  5. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 441
  6. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  7. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  8. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  9. 2022EDGE: Editable Dance Generation From Musiccited by 287
  10. 2023Multisample Flow Matching: Straightening Flows with Minibatch Couplingscited by 278
  11. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  12. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  13. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  14. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 206
  15. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 202
  16. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
  17. 2019Learning Individual Styles of Conversational Gesturecited by 191
  18. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  19. 2022TEACH: Temporal Action Composition for 3D Humanscited by 176
  20. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 165
  21. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  22. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
  23. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  24. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  25. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 98
  26. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 91
  27. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 89
  28. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 86
  29. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  30. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  31. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
  32. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  33. 2025Retargeting Matters: General Motion Retargeting for Humanoid Motion Trackingcited by 75
  34. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  35. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 64
  36. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 58
  37. 2023C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characterscited by 55
  38. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sonic-AMG: Streaming Human Motion Generation from Spatial Audiocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding
  2. 2023Priority-Centric Human Motion Generation in Discrete Latent Space
  3. 2024LEAD: Latent Realignment for Human Motion Diffusion
  4. 2023DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion
  5. 2025MotionGPT3: Human Motion as a Second Modality
  6. 2024VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension
  7. 2023MotionGPT: Human Motion as a Foreign Language
  8. 2025Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions