# Deformable One-shot Face Stylization via DINO Semantic Guidance

arXiv: https://arxiv.org/abs/2403.00459 · HTML: https://mlsanity.com/paper/2403.00459/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Splicing ViT Features for Semantic Appearance Transfer](https://mlsanity.com/paper/2201.00424/) (2022)
- [Few-shot Image Generation via Cross-domain Correspondence](https://mlsanity.com/paper/2104.06820/) (2021)
- [General Facial Representation Learning in a Visual-Linguistic Manner](https://mlsanity.com/paper/2112.03109/) (2021)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains](https://mlsanity.com/paper/2010.05334/) (2020)
- [Improved StyleGAN Embedding: Where are the Good Latents?](https://mlsanity.com/paper/2012.09036/) (2020)
- [Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks](https://mlsanity.com/paper/2110.08398/) (2021)
- [Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer](https://mlsanity.com/paper/2203.13248/) (2022)
- [Few Shot Generative Model Adaption via Relaxed Spatial Structural Alignment](https://mlsanity.com/paper/2203.04121/) (2022)
- [JoJoGAN: One Shot Face Stylization](https://mlsanity.com/paper/2112.11641/) (2021)
- [A Closer Look at Few-shot Image Generation](https://mlsanity.com/paper/2205.03805/) (2022)
- [3DAvatarGAN: Bridging Domains for Personalized Editable Avatars](https://mlsanity.com/paper/2301.02700/) (2023)
- [Few-shot Image Generation via Adaptation-Aware Kernel Modulation](https://mlsanity.com/paper/2210.16559/) (2022)
- [Deformable Style Transfer](https://mlsanity.com/paper/2003.11038/) (2020)
- [One-Shot Adaptation of GAN in Just One CLIP](https://mlsanity.com/paper/2203.09301/) (2022)
- [WarpGAN: Automatic Caricature Generation](https://mlsanity.com/paper/1811.10100/) (2018)

## Built on by (most cited first, 14 of 14)

- [Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensics](https://mlsanity.com/paper/2507.18015/) (2025)
- [UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Training](https://mlsanity.com/paper/2502.02307/) (2025)
- [DiffArtist: Towards Structure and Appearance Controllable Image Stylization](https://mlsanity.com/paper/2407.15842/) (2024)
- [Neural-Driven Image Editing](https://mlsanity.com/paper/2507.05397/) (2025)
- [Domain Generalizable Portrait Style Transfer](https://mlsanity.com/paper/2507.04243/) (2025)
- [ZePo: Zero-Shot Portrait Stylization with Faster Sampling](https://mlsanity.com/paper/2408.05492/) (2024)
- [Latent Space Imaging](https://mlsanity.com/paper/2407.07052/) (2024)
- [FakeRadar: Probing Forgery Outliers to Detect Unknown Deepfake Videos](https://mlsanity.com/paper/2512.14601/) (2025)
- [Real Face Video Animation Platform](https://mlsanity.com/paper/2407.18955/) (2024)
- [AIR: Zero-shot Generative Model Adaptation with Iterative Refinement](https://mlsanity.com/paper/2506.10895/) (2025)
- [One-shot Face Sketch Synthesis in the Wild via Generative Diffusion Prior and Instruction Tuning](https://mlsanity.com/paper/2506.15312/) (2025)
- [CaricatureGS: Exaggerating 3D Gaussian Splatting Faces With Gaussian Curvature](https://mlsanity.com/paper/2601.03319/) (2026)
- [Chameleon: Style-Content Disentangled Framework for Cross-Domain Object Compositing](https://mlsanity.com/paper/2606.01079/) (2026)
- [Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting](https://mlsanity.com/paper/2606.26754/) (2026)

## Abstract (arXiv)

This paper addresses the complex issue of one-shot face stylization, focusing on the simultaneous consideration of appearance and structure, where previous methods have fallen short. We explore deformation-aware face stylization that diverges from traditional single-image style reference, opting for a real-style image pair instead. The cornerstone of our method is the utilization of a self-supervised vision transformer, specifically DINO-ViT, to establish a robust and consistent facial structure representation across both real and style domains. Our stylization process begins by adapting the StyleGAN generator to be deformation-aware through the integration of spatial transformers (STN). We then introduce two innovative constraints for generator fine-tuning under the guidance of DINO semantics: i) a directional deformation loss that regulates directional vectors in DINO space, and ii) a relative structural consistency constraint based on DINO token self-similarities, ensuring diverse generation. Additionally, style-mixing is employed to align the color generation with the reference, minimizing inconsistent correspondences. This framework delivers enhanced deformability for general one-shot face stylization, achieving notable efficiency with a fine-tuning duration of approximately 10 minutes. Extensive qualitative and quantitative comparisons demonstrate our superiority over state-of-the-art one-shot face stylization methods. Code is available at https://github.com/zichongc/DoesFS