# Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation

arXiv: https://arxiv.org/abs/2509.16630 · HTML: https://mlsanity.com/paper/2509.16630/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 106 of its 125 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Builds on (most cited first, 30 of 106)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)

## Built on by (most cited first, 30 of 43)

- [Aurora: Towards Universal Generative Multimodal Time Series Forecasting](https://mlsanity.com/paper/2509.22295/) (2025)
- [FastVMT: Eliminating Redundancy in Video Motion Transfer](https://mlsanity.com/paper/2602.05551/) (2026)
- [Hybrid Token Compression for Vision-Language Models](https://mlsanity.com/paper/2512.08240/) (2025)
- [Follow-Your-Preference: Towards Preference-Aligned Image Inpainting](https://mlsanity.com/paper/2509.23082/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [Top-Down Semantic Refinement for Image Captioning](https://mlsanity.com/paper/2510.22391/) (2025)
- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [Mitty: Diffusion-based Human-to-Robot Video Generation](https://mlsanity.com/paper/2512.17253/) (2025)
- [GCGNet: Graph-Consistent Generative Network for Time Series Forecasting with Exogenous Variables](https://mlsanity.com/paper/2603.08032/) (2026)
- [Group Editing: Edit Multiple Images in One Go](https://mlsanity.com/paper/2603.22883/) (2026)
- [SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation](https://mlsanity.com/paper/2512.12193/) (2025)
- [WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generation](https://mlsanity.com/paper/2511.22098/) (2025)
- [ASTGI: Adaptive Spatio-Temporal Graph Interactions for Irregular Multivariate Time Series Forecasting](https://mlsanity.com/paper/2509.23313/) (2025)
- [CoRA: Boosting Time Series Foundation Models for Multivariate Forecasting through Correlation-aware Adapter](https://mlsanity.com/paper/2603.21828/) (2026)
- [LiveLight: Real-time Streaming Video Relighting with Interactive Control](https://mlsanity.com/paper/2608.01771/) (2026)
- [ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body](https://mlsanity.com/paper/2512.14234/) (2025)
- [Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2603.25155/) (2026)
- [Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers](https://mlsanity.com/paper/2511.18673/) (2025)
- [OmniRefiner: Reinforcement-Guided Local Diffusion Refinement](https://mlsanity.com/paper/2511.19990/) (2025)
- [UIKA: Fast Universal Head Avatar from Pose-Free Images](https://mlsanity.com/paper/2601.07603/) (2026)
- [Embedding-perturbed Exploration Preference Optimization for Flow Models](https://mlsanity.com/paper/2605.15803/) (2026)
- [NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editing](https://mlsanity.com/paper/2511.23105/) (2025)
- [GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflows](https://mlsanity.com/paper/2603.12155/) (2026)
- [MultiMotion: Multi Subject Video Motion Transfer via Video Diffusion Transformer](https://mlsanity.com/paper/2512.07500/) (2025)
- [Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interaction](https://mlsanity.com/paper/2603.21612/) (2026)
- [WDT-MD: Wavelet Diffusion Transformers for Microaneurysm Detection in Fundus Images](https://mlsanity.com/paper/2511.08987/) (2025)
- [AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents](https://mlsanity.com/paper/2602.15325/) (2026)
- [ResWM: Residual-Action World Model for Visual RL](https://mlsanity.com/paper/2603.11110/) (2026)
- [EasyVFX: Frequency-Driven Decoupling for Resource-Efficient VFX Generation](https://mlsanity.com/paper/2605.22051/) (2026)
- [Image Generation as a Visual Planner for Robotic Manipulation](https://mlsanity.com/paper/2512.00532/) (2025)

## Abstract (arXiv)

We present Follow-Your-Emoji-Faster, an efficient diffusion-based framework for freestyle portrait animation driven by facial landmarks. The main challenges in this task are preserving the identity of the reference portrait, accurately transferring target expressions, and maintaining long-term temporal consistency while ensuring generation efficiency. To address identity preservation and accurate expression retargeting, we enhance Stable Diffusion with two key components: a expression-aware landmarks as explicit motion signals, which improve motion alignment, support exaggerated expressions, and reduce identity leakage; and a fine-grained facial loss that leverages both expression and facial masks to better capture subtle expressions and faithfully preserve the reference appearance. With these components, our model supports controllable and expressive animation across diverse portrait types, including real faces, cartoons, sculptures, and animals. However, diffusion-based frameworks typically struggle to efficiently generate long-term stable animation results, which remains a core challenge in this task. To address this, we propose a progressive generation strategy for stable long-term animation, and introduce a Taylor-interpolated cache, achieving a 2.6X lossless acceleration. These two strategies ensure that our method produces high-quality results efficiently, making it user-friendly and accessible. Finally, we introduce EmojiBench++, a more comprehensive benchmark comprising diverse portraits, driving videos, and landmark sequences. Extensive evaluations on EmojiBench++ demonstrate that Follow-Your-Emoji-Faster achieves superior performance in both animation quality and controllability. The code, training dataset and benchmark will be found in https://follow-your-emoji.github.io/.