# Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding

arXiv: https://arxiv.org/abs/2212.02802 · HTML: https://mlsanity.com/paper/2212.02802/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)

## Built on by (most cited first, 30 of 39)

- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder](https://mlsanity.com/paper/2303.17550/) (2023)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [Face Generation and Editing with StyleGAN: A Survey](https://mlsanity.com/paper/2212.09102/) (2022)
- [FinePOSE: Fine-Grained Prompt-Driven 3D Human Pose Estimation via Diffusion Models](https://mlsanity.com/paper/2405.05216/) (2024)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment](https://mlsanity.com/paper/2403.17217/) (2024)
- [RIGID: Recurrent GAN Inversion and Editing of Real Face Videos](https://mlsanity.com/paper/2308.06097/) (2023)
- [CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation](https://mlsanity.com/paper/2407.16193/) (2024)
- [Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates](https://mlsanity.com/paper/2309.11281/) (2023)
- [ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance](https://mlsanity.com/paper/2411.15436/) (2024)
- [Text-driven Visual Synthesis with Latent Diffusion Prior](https://mlsanity.com/paper/2302.08510/) (2023)
- [Video Face Re-Aging: Toward Temporally Consistent Face Re-Aging](https://mlsanity.com/paper/2311.11642/) (2023)
- [Video to Video Generative Adversarial Network for Few-shot Learning Based on Policy Gradient](https://mlsanity.com/paper/2410.20657/) (2024)
- [Detecting Localized Deepfake Manipulations Using Action Unit-Guided Video Representations](https://mlsanity.com/paper/2503.22121/) (2025)
- [DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces](https://mlsanity.com/paper/2509.19230/) (2025)
- [FED-NeRF: Achieve High 3D Consistency and Temporal Coherence for Face Video Editing on Dynamic NeRF](https://mlsanity.com/paper/2401.02616/) (2024)
- [ToonAging: Face Re-Aging upon Artistic Portrait Style Transfer](https://mlsanity.com/paper/2402.02733/) (2024)
- [Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques](https://mlsanity.com/paper/2501.18729/) (2025)
- [ReactDiff: Latent Diffusion for Facial Reaction Generation](https://mlsanity.com/paper/2505.14151/) (2025)
- [S3Editor: A Sparse Semantic-Disentangled Self-Training Framework for Face Video Editing](https://mlsanity.com/paper/2404.08111/) (2024)
- [High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model](https://mlsanity.com/paper/2408.05416/) (2024)
- [FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing](https://mlsanity.com/paper/2505.22141/) (2025)
- [DiViD: Disentangled Video Diffusion for Static-Dynamic Factorization](https://mlsanity.com/paper/2507.13934/) (2025)
- [Durian: Dual Reference Image-Guided Portrait Animation with Attribute Transfer](https://mlsanity.com/paper/2509.04434/) (2025)
- [Revisiting Diffusion Autoencoder Training for Image Reconstruction Quality](https://mlsanity.com/paper/2504.21368/) (2025)
- [Automated Learning of Semantic Embedding Representations for Diffusion Models](https://mlsanity.com/paper/2505.05732/) (2025)

## Abstract (arXiv)

Inspired by the impressive performance of recent face image editing methods, several studies have been naturally proposed to extend these methods to the face video editing task. One of the main challenges here is temporal consistency among edited frames, which is still unresolved. To this end, we propose a novel face video editing framework based on diffusion autoencoders that can successfully extract the decomposed features - for the first time as a face video editing model - of identity and motion from a given video. This modeling allows us to edit the video by simply manipulating the temporally invariant feature to the desired direction for the consistency. Another unique strength of our model is that, since our model is based on diffusion models, it can satisfy both reconstruction and edit capabilities at the same time, and is robust to corner cases in wild face videos (e.g. occluded faces) unlike the existing GAN-based methods.