mlsanityData to 7 Oct 2026

JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model

arXiv 2408.01627 ↗First posted 3 Aug 2024cs.CVcited by 3

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

In recent years, the talking head generation has become a focal point for researchers. Considerable effort is being made to refine lip-sync motion, capture expressive facial expressions, generate natural head poses, and achieve high-quality video. However, no single model has yet achieved equivalence across all quantitative and qualitative metrics. We introduce Jamba, a hybrid Transformer-Mamba model, to animate a 3D face. Mamba, a pioneering Structured State Space Model (SSM) architecture, was developed to overcome the limitations of conventional Transformer architectures, particularly in handling long sequences. This challenge has constrained traditional models. Jamba combines the advantages of both the Transformer and Mamba approaches, offering a comprehensive solution. Based on the foundational Jamba block, we present JambaTalk to enhance motion variety and lip sync through multimodal integration. Extensive experiments reveal that our method achieves performance comparable or superior to state-of-the-art models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 60 of its 79 references that are in the map. First posted on arXiv 3 Aug 2024.

Builds on

60 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models14 Mar

Other references · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  10. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  11. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 267
  12. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
38 more
  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  3. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  4. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  5. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  6. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  7. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  8. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  9. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generationcited by 142
  10. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  11. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
  12. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  13. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  14. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  15. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  16. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  17. 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
  18. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  19. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  20. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  21. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Expertscited by 60
  22. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  23. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  24. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  25. 2023GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generationcited by 42
  26. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  27. 2022Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformerscited by 33
  28. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  29. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  30. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  31. 2024Say Anything with Any Stylecited by 26
  32. 2022Memories are One-to-Many Mapping Alleviators in Talking Face Generationcited by 22
  33. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 22
  34. 2023Towards Realistic Generative 3D Face Modelscited by 20
  35. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  36. 2019Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Netscited by 19
  37. 2021Sparse to Dense Dynamic 3D Facial Expression Generationcited by 18
  38. 2022Talking Face Generation with Multilingual TTScited by 18

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EditEmoTalk: Controllable Speech-Driven 3D Facial Animation with Continuous Expression Editingcited by 1
  2. 2026PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordinationcited by 0
  3. 2026SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generation
  2. 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation
  3. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
  4. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation
  5. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusion
  6. 2024GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer
  7. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  8. 2023From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications