# EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions

arXiv: https://arxiv.org/abs/2407.08136 · HTML: https://mlsanity.com/paper/2407.08136/

## Key facts

- Cited by 196 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation](https://mlsanity.com/paper/2406.08801/) (2024)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2406.01900/) (2024)
- [MegaPortraits: One-shot Megapixel Neural Head Avatars](https://mlsanity.com/paper/2207.07621/) (2022)
- [V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation](https://mlsanity.com/paper/2406.02511/) (2024)
- [VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior](https://mlsanity.com/paper/2312.01841/) (2023)
- [Human MotionFormer: Transferring Human Motions with Vision Transformers](https://mlsanity.com/paper/2302.11306/) (2023)

## Built on by (most cited first, 30 of 196)

- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency](https://mlsanity.com/paper/2409.02634/) (2024)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation](https://mlsanity.com/paper/2410.07718/) (2024)
- [OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation](https://mlsanity.com/paper/2506.18866/) (2025)
- [FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis](https://mlsanity.com/paper/2504.04842/) (2025)
- [EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation](https://mlsanity.com/paper/2411.10061/) (2024)
- [Sonic: Shifting Focus to Global Audio Perception in Portrait Animation](https://mlsanity.com/paper/2411.16331/) (2024)
- [Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation](https://mlsanity.com/paper/2505.22647/) (2025)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation](https://mlsanity.com/paper/2503.18860/) (2025)
- [LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision](https://mlsanity.com/paper/2412.09262/) (2024)
- [Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model](https://mlsanity.com/paper/2502.09533/) (2025)
- [StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation](https://mlsanity.com/paper/2508.08248/) (2025)
- [Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2509.16630/) (2025)
- [Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis](https://mlsanity.com/paper/2411.19509/) (2024)
- [InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing](https://mlsanity.com/paper/2508.14033/) (2025)
- [FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait](https://mlsanity.com/paper/2412.01064/) (2024)
- [INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations](https://mlsanity.com/paper/2412.04037/) (2024)
- [CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention](https://mlsanity.com/paper/2409.01876/) (2024)
- [MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation](https://mlsanity.com/paper/2412.04448/) (2024)
- [SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers](https://mlsanity.com/paper/2502.10841/) (2025)
- [OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation](https://mlsanity.com/paper/2508.19209/) (2025)
- [EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation](https://mlsanity.com/paper/2507.03905/) (2025)
- [MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling](https://mlsanity.com/paper/2410.10122/) (2024)
- [X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention](https://mlsanity.com/paper/2507.23143/) (2025)
- [OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers](https://mlsanity.com/paper/2505.21448/) (2025)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars](https://mlsanity.com/paper/2512.22065/) (2025)

## Abstract (arXiv)

The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.