mlsanityData to 7 Oct 2026

Talking Head(?) Anime from a Single Image 4: Improved Model and Its Distillation

arXiv 2311.17409 ↗First posted 29 Nov 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the problem of creating a character model that can be controlled in real time from a single image of an anime character. A solution to this problem would greatly reduce the cost of creating avatars, computer games, and other interactive applications. Talking Head Anime 3 (THA3) is an open source project that attempts to directly address the problem. It takes as input (1) an image of an anime character's upper body and (2) a 45-dimensional pose vector and outputs a new image of the same character taking the specified pose. The range of possible movements is expressive enough for personal avatars and certain types of game characters. However, the system is too slow to generate animations in real time on common PCs, and its image quality can be improved. In this paper, we improve THA3 in two ways. First, we propose new architectures for constituent networks that rotate the character's head and body based on U-Nets with attention that are widely used in modern generative models. The new architectures consistently yield better image quality than the THA3 baseline. Nevertheless, they also make the whole system much slower: it takes up to 150 milliseconds to generate a frame. Second, we propose a technique to distill the system into a small network (less than 2 MB) that can generate 512x512 animation frames in real time (under 30 FPS) using consumer gaming GPUs while keeping the image quality close to that of the full system. This improvement makes the whole system practical for real-time applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 57 of its 84 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

57 of 84 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
38 more
  1. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  2. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  3. 2014Conditional Generative Adversarial Netscited by 2,152
  4. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  8. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  9. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  10. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  11. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  12. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  13. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  14. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  15. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 550
  16. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  17. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  18. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  19. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  20. 2021Animatable Neural Radiance Fields for Modeling Dynamic Human Bodiescited by 361
  21. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  22. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  23. 2022BBDM: Image-to-image Translation with Brownian Bridge Diffusion Modelscited by 222
  24. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  25. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  26. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  27. 2020STAR: Sparse Trained Articulated Human Body Regressorcited by 194
  28. 2021ARCH++: Animation-Ready Clothed Human Reconstruction Revisitedcited by 174
  29. 2016View Synthesis by Appearance Flowcited by 172
  30. 2019NASA: Neural Articulated Shape Approximationcited by 149
  31. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  32. 2020i3DMM: Deep Implicit 3D Morphable Model of Human Headscited by 109
  33. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  34. 2022ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representationscited by 64
  35. 2020CONFIG: Controllable Neural Face Image Generationcited by 61
  36. 2017Transformation-Grounded Image Generation Network for Novel 3D View Synthesiscited by 59
  37. 20193D Guided Fine-Grained Face Manipulationcited by 45
  38. 2023FitMe: Deep Photorealistic 3D Morphable Model Avatarscited by 38

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.