mlsanityData to 14 Oct 2026

MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation

arXiv 2412.04448 ↗First posted 5 Dec 2024cs.CVcited by 38

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advances in video diffusion models have unlocked new potential for realistic audio-driven talking video generation. However, achieving seamless audio-lip synchronization, maintaining long-term identity consistency, and producing natural, audio-aligned expressions in generated talking videos remain significant challenges. To address these challenges, we propose Memory-guided EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) a memory-guided temporal module, which enhances long-term identity consistency and motion smoothness by developing memory states to store information from a longer past context to guide temporal modeling via linear attention; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion adaptive layer norm. Extensive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 49 of its 76 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

49 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation10 Oct
  2. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency4 Sep
  3. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan

Other references · 45

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
33 more
  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  4. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  5. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  6. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  7. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  8. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  9. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  10. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  11. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  12. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  13. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  14. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 176
  15. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  16. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  17. 2018Lip Movements Generation at a Glancecited by 155
  18. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 121
  19. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  20. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 86
  21. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 62
  22. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
  23. 2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
  24. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  25. 2023VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Priorcited by 47
  26. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  27. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 40
  28. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantizationcited by 39
  29. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  30. 2018Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languagescited by 25
  31. 2023HiPA: Enabling One-Step Text-to-Image Diffusion Models via High-Frequency-Promoting Adaptationcited by 16
  32. 2023EMNS /Imz/ Corpus: An emotive single-speaker dataset for narrative storytelling in games, television and graphic novelscited by 7
  33. 2022A New Amharic Speech Emotion Dataset and Classification Benchmarkcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 186
  2. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  3. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 59
  4. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 35
  5. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  6. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  7. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  8. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  9. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 14
  10. 2025DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generationcited by 13
  11. 2025SlideTailor: Personalized Presentation Slide Generation for Scientific Paperscited by 13
  12. 2025Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generationcited by 10
26 more
  1. 2025MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmarkcited by 9
  2. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  3. 2025Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedbackcited by 8
  4. 2026AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoisingcited by 6
  5. 2025RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformercited by 5
  6. 2025PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysiscited by 5
  7. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  8. 2025PESTalk: Speech-Driven 3D Facial Animation with Personalized Emotional Stylescited by 4
  9. 2025X-Actor: Emotional and Expressive Long-Range Portrait Acting from Audiocited by 3
  10. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformerscited by 2
  11. 2025Talking Head Generation via AU-Guided Landmark Predictioncited by 1
  12. 2025Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Datacited by 1
  13. 2026AUHead: Realistic Emotional Talking Head Generation via Action Units Controlcited by 1
  14. 2026FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generationcited by 1
  15. 2025Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generationcited by 0
  16. 2025YingVideo-MV: Music-Driven Multi-Stage Video Generationcited by 0
  17. 2025A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniquescited by 0
  18. 2026The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detectioncited by 0
  19. 2026From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detectioncited by 0
  20. 2026Temporally-Aligned Evaluation for Audio-Driven Talking Head Generationcited by 0
  21. 2026CUPID: Reconstructing UV Texture Maps for Interpretable Person-of-Interest Deepfake Detectioncited by 0
  22. 2026Foundation Models are Implicit Deepfake Detectorscited by 0
  23. 2026Xemo-Talker: Unlock Emotions Explicitly for Audio-Driven Talking Portrait Synthesiscited by 0
  24. 2026AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Modelcited by 0
  25. 2026DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Expertscited by 0
  26. 2026Decoupled Self-Forcing Distillation for Streaming Talking Head Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion
  2. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
  3. 2024DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis
  4. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
  5. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  6. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
  7. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
  8. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion