CatPortrait: Taming Video Generative Models for Portrait Animation via In-context Conditioning
Not on the map. Why?
Abstract (arXiv)
Recent video generative models have shown strong capability in synthesizing high-quality and temporally coherent videos. Yet, adapting them for controllable portrait animation remains nontrivial, as existing methods typically rely on architectural changes or extra conditioning modules, increasing complexity and reducing compatibility with pretrained models. We present CatPortrait, a simple and effective framework that enables controllable portrait animation via 3D face guidance as in-context conditioning. CatPortrait preserves the original video generation backbone and injects structured 3D facial guidance into the input space through spatial concatenation, allowing pretrained image-to-video models to animate portraits without sacrificing their pretrained priors. CatPortrait supports both video-driven and audio-driven portrait animation within a unified framework. Extensive experiments demonstrate that CatPortrait achieves competitive or superior results across both settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 10 Oct 2026.
Builds on
42 of 53 references.
References · 42
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2022Human Motion Diffusion Modelcited by 1,080
- 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
- 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
- 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 444
- 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
30 more
- 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
- 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
- 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
- 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
- 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
- 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 178
- 2018Lip Movements Generation at a Glancecited by 155
- 2024CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Modelscited by 102
- 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
- 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
- 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 91
- 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 84
- 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 80
- 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 74
- 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 59
- 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
- 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
- 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 47
- 2025AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Modelscited by 14
- 2024Video Diffusion Transformers are In-Context Learnerscited by 12
- 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
- 2023DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarkscited by 10
- 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generationcited by 10
- 2026TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generationcited by 7
- 2025VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioningcited by 5
- 2025MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidancecited by 4
- 2025FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesiscited by 3
- 2025Dual Audio-Centric Modality Coupling for Talking Head Generationcited by 3
- 2026Reward-Aware Trajectory Shaping for Few-step Visual Generationcited by 2
- 2026Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generationcited by 2
Built on by
Not cited yet.