# Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation

arXiv: https://arxiv.org/abs/2606.03402 · HTML: https://mlsanity.com/paper/2606.03402/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 23 of its 24 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)

## Builds on (most cited first, 23 of 23)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Thin-Plate Spline Motion Model for Image Animation](https://mlsanity.com/paper/2203.14367/) (2022)
- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation](https://mlsanity.com/paper/2410.07718/) (2024)
- [X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention](https://mlsanity.com/paper/2403.15931/) (2024)
- [Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation](https://mlsanity.com/paper/2307.09906/) (2023)
- [Audio-Driven Co-Speech Gesture Video Generation](https://mlsanity.com/paper/2212.02350/) (2022)
- [VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis](https://mlsanity.com/paper/2403.08764/) (2024)
- [DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder](https://mlsanity.com/paper/2303.17550/) (2023)
- [Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model](https://mlsanity.com/paper/2404.01862/) (2024)
- [TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation](https://mlsanity.com/paper/2410.04221/) (2024)
- [DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures](https://mlsanity.com/paper/2409.07649/) (2024)
- [Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models](https://mlsanity.com/paper/2507.20220/) (2025)

## Abstract (arXiv)

Audio-driven human motion video generation aims to synthesize realistic and temporally coherent human animations from a single static image, with applications in talking-head synthesis, co-speech gesture generation, and dynamic presentations. Moving beyond conventional keypoint-based methods that often struggle to capture subtle motion dynamics, We propose a novel implicit-motion framework for generating realistic and temporally coherent human motion videos from a single static image and audio. Our approach uses a two-stage pipeline that decouples motion prediction from rendering. The first stage integrates appearance priors and hierarchical depth cues into a region-aware attention mechanism to model latent motion features. The second stage employs a Mamba-enhanced diffusion model to directly predict these features from audio and the source image, enabling unsupervised learning of fine-grained motion patterns. This decoupled architecture enhances flexibility and efficiency. Trained on a new 380-hour high-quality dataset, our method outperforms prior work across multiple public benchmarks and our collected data in accuracy, naturalness, and temporal coherence, setting a new state-of-the-art.