mlsanityData to 7 Oct 2026

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

arXiv 2401.08503 ↗First posted 16 Jan 2024cs.CVcited by 97

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Besides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3D-Potrait, a framework that (1) improves the one-shot 3D reconstruction power with a large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-conditioned animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using a head-torso-background super-resolution model; and (4) supports one-shot audio-driven talking face generation with a generalizable audio-to-motion model. Extensive experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos compared to previous methods. Video samples and source code are available at https://real3dportrait.github.io .

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

48 of 57 references.

References · 48

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  8. 2020First Order Motion Model for Image Animationcited by 698
  9. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  10. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  11. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  12. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
36 more
  1. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  2. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  3. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  4. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  5. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  6. 2018Deep Video Portraitscited by 265
  7. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  8. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  9. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  10. 2021HeadNeRF: A Real-time NeRF-based Parametric Head Modelcited by 228
  11. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  12. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 177
  13. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  14. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  15. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  16. 2022Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatarscited by 124
  17. 2017You said that?cited by 99
  18. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  19. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
  20. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  21. 2023One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Fieldcited by 68
  22. 2023OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Renderingcited by 67
  23. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  24. 2023Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generationcited by 61
  25. 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
  26. 2023NOFA: NeRF-based One-shot Facial Avatar Reconstructioncited by 53
  27. 2019Neural Style-Preserving Visual Dubbingcited by 47
  28. 2023DPE: Disentanglement of Pose and Expression for General Video Portrait Editingcited by 45
  29. 2020What comprises a good talking-head video generation?: A Survey and Benchmarkcited by 42
  30. 2023GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generationcited by 42
  31. 20223D GAN Inversion with Facial Symmetry Priorcited by 39
  32. 2022Audio-Driven Talking Face Video Generation with Dynamic Convolution Kernelscited by 23
  33. 2023Generalizable One-shot Neural Head Avatarcited by 21
  34. 2023Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesiscited by 8
  35. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  36. 2023OPHAvatars: One-shot Photo-realistic Head Avatarscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 44; 2026: 24.

97 papers.

Sort:

By area:

  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  3. 2024Generalizable and Animatable Gaussian Head Avatarcited by 95
  4. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  5. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  6. 2024Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizercited by 64
  7. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
  8. 2024CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Modelscited by 52
  9. 2025Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Modelcited by 49
  10. 2024HeadGAP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priorscited by 44
  11. 2025LAM: Large Avatar Model for One-shot Animatable Gaussian Headcited by 43
  12. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
38 more
  1. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  2. 2025ARTalk: Speech-Driven 3D Head Animation via Autoregressive Modelcited by 28
  3. 2025Human Motion Video Generation: A Surveycited by 25
  4. 2024VOODOO XP: Expressive One-Shot Head Reenactment for VR Telepresencecited by 19
  5. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  6. 2025Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensicscited by 18
  7. 2024Joker: Conditional 3D Head Synthesis with Extreme Facial Expressionscited by 17
  8. 2024EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Headcited by 16
  9. 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
  10. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  11. 2025AvatarArtist: Open-Domain 4D Avatarizationcited by 15
  12. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformerscited by 13
  13. 2024MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devicescited by 12
  14. 2025SEGA: Drivable 3D Gaussian Head Avatar from a Single Imagecited by 11
  15. 2025Monocular and Generalizable Gaussian Talking Head Animationcited by 10
  16. 2025FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervisioncited by 10
  17. 2024DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generationcited by 8
  18. 2025FastAvatar: Instant 3D Gaussian Splatting for Faces from Single Unconstrained Posescited by 7
  19. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  20. 2025FaceCraft4D: Animated 3D Facial Avatar Generation from a Single Imagecited by 5
  21. 2025Audio Driven Real-Time Facial Animation for Social Telepresencecited by 5
  22. 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
  23. 2026UIKA: Fast Universal Head Avatar from Pose-Free Imagescited by 5
  24. 2024PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesiscited by 4
  25. 2024FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillationcited by 4
  26. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generationcited by 4
  27. 2025GHOST 2.0: generative high-fidelity one shot transfer of headscited by 4
  28. 2025GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptationcited by 4
  29. 2024Coherent 3D Portrait Video Reconstruction via Triplane Fusioncited by 3
  30. 2024Text-based Talking Video Editing with Cascaded Conditional Diffusioncited by 3
  31. 2024JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Modelcited by 3
  32. 2024IF-MDM: Implicit Face Motion Diffusion Model for High-Fidelity Realtime Talking Head Generationcited by 3
  33. 2025Identity-Preserving Video Dubbing Using Motion Warpingcited by 3
  34. 2025Controlling Avatar Diffusion with Learnable Gaussian Embeddingcited by 3
  35. 2025EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameterscited by 3
  36. 2025DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animationcited by 3
  37. 2025VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Imagecited by 3
  38. 2025Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamicscited by 3

50 most cited of 97 All 97 (JSON)