mlsanityData to 7 Oct 2026

Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation

arXiv 2406.01900 ↗First posted 4 Jun 2024cs.CVcited by 172

Not on the map. Why?

Cited by 50 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences. The main challenge of portrait animation is to preserve the identity of the reference portrait and transfer the target expression to this portrait while maintaining temporal consistency and fidelity. To address these challenges, Follow-Your-Emoji equipped the powerful Stable Diffusion model with two well-designed technologies. Specifically, we first adopt a new explicit motion signal, namely expression-aware landmark, to guide the animation process. We discover this landmark can not only ensure the accurate motion alignment between the reference portrait and target motion during inference but also increase the ability to portray exaggerated expressions (i.e., large pupil movements) and avoid identity leakage. Then, we propose a facial fine-grained loss to improve the model's ability of subtle expression perception and reference portrait appearance reconstruction by using both expression and facial masks. Accordingly, our method demonstrates significant performance in controlling the expression of freestyle portraits, including real humans, cartoons, sculptures, and even animals. By leveraging a simple and effective progressive generation strategy, we extend our model to stable long-term animation, thus increasing its potential application value. To address the lack of a benchmark for this field, we introduce EmojiBench, a comprehensive benchmark comprising diverse portrait images, driving videos, and landmarks. We show extensive evaluations on EmojiBench to verify the superiority of Follow-Your-Emoji.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Builds on 75 of its 92 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

75 of 92 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2022Video Diffusion Modelscited by 2,280
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  10. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  11. 2020First Order Motion Model for Image Animationcited by 698
  12. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  13. 2023ModelScope Text-to-Video Technical Reportcited by 651
  14. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  15. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  16. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  17. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  18. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  19. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  20. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  21. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  22. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  23. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  24. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  25. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  26. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  27. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  28. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  29. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  30. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  31. 2018Deep Video Portraitscited by 265
  32. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  33. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  34. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  35. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  36. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  37. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  38. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 93; 2026: 49.

172 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  3. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  4. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  5. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  6. 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 100
  7. 2025Controllable Video Generation: A Surveycited by 93
  8. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  9. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
  10. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  11. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  12. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
38 more
  1. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  2. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 55
  3. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  4. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  5. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
  6. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 47
  7. 2025EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editingcited by 45
  8. 2024Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generationcited by 42
  9. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  10. 2024FonTS: Text Rendering with Typography and Style Controlscited by 40
  11. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  12. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  13. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 35
  14. 2024InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differencescited by 34
  15. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
  16. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  17. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecastingcited by 28
  18. 2025Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Controlcited by 27
  19. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
  20. 2025X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attentioncited by 26
  21. 2025EasyText: Controllable Diffusion Transformer for Multilingual Text Renderingcited by 25
  22. 2025Human Motion Video Generation: A Surveycited by 25
  23. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  24. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  25. 2024AniDoc: Animation Creation Made Easiercited by 22
  26. 2025DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulationcited by 21
  27. 2024Large Motion Video Autoencoding with Cross-modal Video VAEcited by 19
  28. 2025Hybrid Token Compression for Vision-Language Modelscited by 19
  29. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  30. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  31. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  32. 2025MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatarscited by 17
  33. 2024DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Posescited by 16
  34. 2024Towards Multiple Character Image Animation Through Enhancing Implicit Decouplingcited by 15
  35. 2024ReFIR: Grounding Large Restoration Models with Retrieval Augmentationcited by 15
  36. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  37. 2025LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correctioncited by 15
  38. 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Stable Video-Driven Portraits
  2. 2024AniFaceDiff: Animating Stylized Avatars via Parametric Conditioned Diffusion Models
  3. 2024Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
  4. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
  5. 2026MultiAnimate: Pose-Guided Image Animation Made Extensible
  6. 2025SkyReels-A1: Expressive Portrait Animation in Video Diffusion Transformers
  7. 2025ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model
  8. 2025Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidance