mlsanityData to 7 Oct 2026

MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models

arXiv 2403.09471 ↗First posted 14 Mar 2024cs.CV cs.HCcited by 46

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. Recent advancements have utilized the diffusion model and attention mechanisms to improve gesture synthesis. However, due to the high computational complexity of these techniques, generating long and diverse sequences with low latency remains a challenge. We explore the potential of state space models (SSMs) to address the challenge, implementing a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures. Leveraging the foundational Mamba block, we introduce MambaTalk, enhancing gesture diversity and rhythm through multimodal integration. Extensive experiments demonstrate that our method matches or exceeds the performance of state-of-the-art models. Our project is publicly available at https://kkakkkka.github.io/MambaTalk

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 60 of its 84 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

60 of 84 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2024VMamba: Visual State Space Modelcited by 955
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
38 more
  1. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  3. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  5. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  6. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  7. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  8. 2022Long Range Language Modeling via Gated State Spacescited by 265
  9. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  10. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  11. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  12. 2019Learning Individual Styles of Conversational Gesturecited by 190
  13. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  14. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  15. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  16. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  17. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  18. 2022Liquid Structural State-Space Modelscited by 134
  19. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
  20. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  21. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  22. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  23. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  24. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
  25. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  26. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  27. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  28. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  29. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priorscited by 64
  30. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  31. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  32. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  33. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  34. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  35. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  36. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  37. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  38. 2023G2L: Semantically Aligned and Uniform Video Grounding via Geodesic and Game Theorycited by 21

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 18; 2026: 13.

46 papers.

Sort:

By area:

  1. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  2. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 55
  3. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  4. 2024A Survey on Visual Mambacited by 50
  5. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  6. 2024DisPose: Disentangling Pose Guidance for Controllable Human Image Animationcited by 35
  7. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  8. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  9. 2024Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Networkcited by 26
  10. 2024MVGamba: Unify 3D Content Generation as State Space Sequence Modelingcited by 23
  11. 2024The Evolution of Multimodal Model Architecturescited by 21
  12. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
34 more
  1. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  2. 2024MambaCSR: Dual-Interleaved Scanning for Compressed Image Super-Resolution With SSMscited by 14
  3. 2025MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animationcited by 14
  4. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  5. 2024Alignment is All You Need: A Training-free Augmentation Strategy for Pose-guided Video Generationcited by 10
  6. 2025FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generationcited by 10
  7. 2025MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generationcited by 9
  8. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  9. 2025Intentional Gesture: Deliver Your Intentions with Gestures for Speechcited by 8
  10. 2024AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Rewardcited by 7
  11. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  12. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  13. 2026MultiAnimate: Pose-Guided Image Animation Made Extensiblecited by 6
  14. 2024A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motionscited by 4
  15. 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesiscited by 4
  16. 2025InterAnimate: Taming Region-aware Diffusion Model for Realistic Human Interaction Animationcited by 4
  17. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  18. 2026LiveGesture Streamable Co-Speech Gesture Generation Modelcited by 4
  19. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3
  20. 2025InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generationcited by 3
  21. 2026TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mambacited by 3
  22. 2025ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesiscited by 2
  23. 2025Creative4U: MLLMs-based Advertising Creative Image Selector with Comparative Reasoningcited by 2
  24. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mambacited by 1
  25. 2025MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantizationcited by 1
  26. 20263DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Controlcited by 1
  27. 2026PersonaGest: Personalized Co-Speech Gesture Generation with Semantic-Guided Hierarchical Motion Representationcited by 1
  28. 2026DuoGesture: Motion-Grounded Semantic Conditioning and Biomechanical Beat Priors for Co-Speech Gesture Generationcited by 1
  29. 2025Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learningcited by 0
  30. 2026SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesturecited by 0
  31. 2026StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoringcited by 0
  32. 2026Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialoguecited by 0
  33. 2026Multi-Modal Controlled Coherent Motion Generationcited by 0
  34. 2026SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MambaGesture: Enhancing Co-Speech Gesture Generation with Mamba and Disentangled Multi-Modality Fusion
  2. 2025Dyadic Mamba: Long-term Dyadic Human Motion Synthesis
  3. 2025InterMamba: Efficient Human-Human Interaction Generation with Adaptive Spatio-Temporal Mamba
  4. 2025MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition
  5. 2024Motion Mamba: Efficient and Long Sequence Motion Generation
  6. 2023DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
  7. 2024DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework
  8. 2024Large Body Language Models