# FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning

arXiv: https://arxiv.org/abs/2312.12142 · HTML: https://mlsanity.com/paper/2312.12142/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2205.09542/) (2022)
- [DG-Font: Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2104.03064/) (2021)
- [Separating Style and Content for Generalized Style Transfer](https://mlsanity.com/paper/1711.06454/) (2017)
- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [Few-shot Font Generation with Localized Style Representations and Factorization](https://mlsanity.com/paper/2009.11042/) (2020)
- [Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts](https://mlsanity.com/paper/2104.00887/) (2021)
- [Diff-Font: Diffusion Model for Robust One-Shot Font Generation](https://mlsanity.com/paper/2212.05895/) (2022)
- [Few-Shot Font Generation by Learning Fine-Grained Local Styles](https://mlsanity.com/paper/2205.09965/) (2022)
- [Look Closer to Supervise Better: One-Shot Font Generation via Component-Based Discriminator](https://mlsanity.com/paper/2205.00146/) (2022)
- [Auto-Encoder Guided GAN for Chinese Calligraphy Synthesis](https://mlsanity.com/paper/1706.08789/) (2017)
- [Learning to Write Stylized Chinese Characters by Reading a Handful of Examples](https://mlsanity.com/paper/1712.06424/) (2017)
- [CalliGAN: Style and Structure-aware Chinese Calligraphy Character Generator](https://mlsanity.com/paper/2005.12500/) (2020)

## Built on by (most cited first, 30 of 48)

- [AnyText2: Visual Text Generation and Editing With Customizable Attributes](https://mlsanity.com/paper/2411.15245/) (2024)
- [Visual Text Generation in the Wild](https://mlsanity.com/paper/2407.14138/) (2024)
- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [Deciphering Oracle Bone Language with Diffusion Models](https://mlsanity.com/paper/2406.00684/) (2024)
- [DiffusionPen: Towards Controlling the Style of Handwritten Text Generation](https://mlsanity.com/paper/2409.06065/) (2024)
- [Hand1000: Generating Realistic Hands from Text with Only 1,000 Images](https://mlsanity.com/paper/2408.15461/) (2024)
- [Kinetic Typography Diffusion Model](https://mlsanity.com/paper/2407.10476/) (2024)
- [Predicting the Original Appearance of Damaged Historical Documents](https://mlsanity.com/paper/2412.11634/) (2024)
- [Decoupling Layout from Glyph in Online Chinese Handwriting Generation](https://mlsanity.com/paper/2410.02309/) (2024)
- [Design-o-meter: Towards Evaluating and Refining Graphic Designs](https://mlsanity.com/paper/2411.14959/) (2024)
- [GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models](https://mlsanity.com/paper/2408.07259/) (2024)
- [Font Style Interpolation with Diffusion Models](https://mlsanity.com/paper/2402.14311/) (2024)
- [CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control](https://mlsanity.com/paper/2410.03787/) (2024)
- [MegaHan97K: A Large-Scale Dataset for Mega-Category Chinese Character Recognition with over 97K Categories](https://mlsanity.com/paper/2506.04807/) (2025)
- [Calligrapher: Freestyle Text Image Customization](https://mlsanity.com/paper/2506.24123/) (2025)
- [VecGlypher: Unified Vector Glyph Generation with Language Models](https://mlsanity.com/paper/2602.21461/) (2026)
- [MX-Font++: Mixture of Heterogeneous Aggregation Experts for Few-shot Font Generation](https://mlsanity.com/paper/2503.02799/) (2025)
- [UniCalli: A Unified Diffusion Framework for Column-Level Generation and Recognition of Chinese Calligraphy](https://mlsanity.com/paper/2510.13745/) (2025)
- [TextWand: A Unified Framework for Scene Text Editing](https://mlsanity.com/paper/2606.05730/) (2026)
- [SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild](https://mlsanity.com/paper/2501.02962/) (2025)
- [FontAdapter: Instant Font Adaptation in Visual Text Generation](https://mlsanity.com/paper/2506.05843/) (2025)
- [Embedding Font Impression Word Tags Based on Co-occurrence](https://mlsanity.com/paper/2508.18825/) (2025)
- [DA-Font: Few-Shot Font Generation via Dual-Attention Hybrid Integration](https://mlsanity.com/paper/2509.16632/) (2025)
- [Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation](https://mlsanity.com/paper/2601.01593/) (2026)
- [CONSTANT: Towards High-Quality One-Shot Handwriting Generation with Patch Contrastive Enhancement and Style-Aware Quantization](https://mlsanity.com/paper/2603.07543/) (2026)
- [FontCrafter: High-Fidelity Element-Driven Artistic Font Creation with Visual In-Context Generation](https://mlsanity.com/paper/2603.22054/) (2026)
- [Cross-Domain Image Conversion by CycleDM](https://mlsanity.com/paper/2403.02919/) (2024)
- [Total Disentanglement of Font Images into Style and Character Class Features](https://mlsanity.com/paper/2403.12784/) (2024)
- [One-Shot Multilingual Font Generation Via ViT](https://mlsanity.com/paper/2412.11342/) (2024)

## Abstract (arXiv)

Automatic font generation is an imitation task, which aims to create a font library that mimics the style of reference images while preserving the content from source images. Although existing font generation methods have achieved satisfactory performance, they still struggle with complex characters and large style variations. To address these issues, we propose FontDiffuser, a diffusion-based image-to-image one-shot font generation method, which innovatively models the font imitation task as a noise-to-denoise paradigm. In our method, we introduce a Multi-scale Content Aggregation (MCA) block, which effectively combines global and local content cues across different scales, leading to enhanced preservation of intricate strokes of complex characters. Moreover, to better manage the large variations in style transfer, we propose a Style Contrastive Refinement (SCR) module, which is a novel structure for style representation learning. It utilizes a style extractor to disentangle styles from images, subsequently supervising the diffusion model via a meticulously designed style contrastive loss. Extensive experiments demonstrate FontDiffuser's state-of-the-art performance in generating diverse characters and styles. It consistently excels on complex characters and large style changes compared to previous methods. The code is available at https://github.com/yeungchenwa/FontDiffuser.