mlsanityData to 7 Oct 2026

HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping

arXiv 2212.06458 ↗First posted 13 Dec 2022cs.CVcited by 12

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Image-based head swapping task aims to stitch a source head to another source body flawlessly. This seldom-studied task faces two major challenges: 1) Preserving the head and body from various sources while generating a seamless transition region. 2) No paired head swapping dataset and benchmark so far. In this paper, we propose a semantic-mixing diffusion model for head swapping (HS-Diffusion) which consists of a latent diffusion model (LDM) and a semantic layout generator. We blend the semantic layouts of source head and source body, and then inpaint the transition region by the semantic layout generator, achieving a coarse-grained head swapping. Semantic-mixing LDM can further implement a fine-grained head swapping with the inpainted layout as condition by a progressive fusion process, while preserving head and body with high-quality reconstruction. To this end, we propose a semantic calibration strategy for natural inpainting and a neck alignment for geometric realism. Importantly, we construct a new image-based head swapping benchmark and design two tailor-designed metrics (Mask-FID and Focal-FID). Extensive experiments demonstrate the superiority of our framework. The code will be available: https://github.com/qinghew/HS-Diffusion.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 56 of its 71 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

56 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Blended Latent Diffusion6 Jun
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
38 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  2. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  3. 2016Least Squares Generative Adversarial Networkscited by 1,000
  4. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  5. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  6. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  7. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  8. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  9. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  10. 2021GAN Inversion: A Surveycited by 366
  11. 2017VITON: An Image-based Virtual Try-on Networkcited by 285
  12. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  13. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  14. 2021SimSwap: An Efficient Framework For High Fidelity Face Swappingcited by 243
  15. 2022Diffusion models as plug-and-play priorscited by 230
  16. 2019FaceShifter: Towards High Fidelity And Occlusion Aware Face Swappingcited by 221
  17. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  18. 2019Self-Correction for Human Parsingcited by 179
  19. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  20. 2020Towards Photo-Realistic Virtual Try-On by Adaptively GeneratingleftrightarrowPreserving Image Contentcited by 148
  21. 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 143
  22. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  23. 2020DeepFaceLab: Integrated, flexible and extensible face-swapping frameworkcited by 129
  24. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  25. 2021One Shot Face Swapping on Megapixelscited by 105
  26. 2020A Survey on Generative Adversarial Networks: Variants, Applications, and Trainingcited by 97
  27. 2021Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editingcited by 91
  28. 2021PD-GAN: Probabilistic Diverse GAN for Image Inpaintingcited by 80
  29. 2022High-resolution Face Swapping via Latent Semantics Disentanglementcited by 57
  30. 2021Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGANcited by 45
  31. 2022Region-Aware Face Swappingcited by 45
  32. 2022InsetGAN for Full-Body Image Generationcited by 45
  33. 2019Breaking the cycle -- Colleagues are all you needcited by 40
  34. 2021PaddleSeg: A High-Efficient Development Toolkit for Image Segmentationcited by 33
  35. 2021Toward Accurate and Realistic Outfits Visualization with Attention to Detailscited by 33
  36. 2021Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothnesscited by 30
  37. 20193DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translationcited by 27
  38. 2021FaceController: Controllable Attribute Editing for Face in the Wildcited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 2.

12 papers.

  1. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  2. 2024Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Modelscited by 35
  3. 2024CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Modelscited by 24
  4. 2023A Generalist FaceX via Learning Unified Facial Representationcited by 13
  5. 2024Training-Free Semantic Video Composition via Pre-trained Diffusion Modelcited by 4
  6. 2025GHOST 2.0: generative high-fidelity one shot transfer of headscited by 4
  7. 2025Zero-Shot Head Swapping in Real-World Scenarioscited by 4
  8. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  9. 2025Controllable and Expressive One-Shot Video Head Swappingcited by 2
  10. 2026Towards High Fidelity Face Swapping: A Comprehensive Survey and New Benchmarkcited by 1
  11. 2025DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluationcited by 0
  12. 2026UniBioTransfer: A Unified Framework for Multiple Biometrics Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022DiffFace: Diffusion-based Face Swapping with Facial Guidance
  2. 2024Face Swap via Diffusion Model
  3. 2026VFace: A Training-Free Approach for Diffusion-Based Video Face Swapping
  4. 2025High-Fidelity Diffusion Face Swapping with ID-Constrained Facial Conditioning
  5. 2024VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping
  6. 2026Edit3DGS: Unified Framework for Dynamic Head Editing via 2D Instruction-Guided Diffusion and 3D Gaussian Splatting
  7. 2024Controllable Face Synthesis with Semantic Latent Diffusion Models
  8. 2026An Evaluation Framework for Generating Multi-View Images of a Person in a Scene