# XMP-Font: Self-Supervised Cross-Modality Pre-training for Few-Shot Font Generation

arXiv: https://arxiv.org/abs/2204.05084 · HTML: https://mlsanity.com/paper/2204.05084/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Universal Style Transfer via Feature Transforms](https://mlsanity.com/paper/1705.08086/) (2017)
- [DualGAN: Unsupervised Dual Learning for Image-to-Image Translation](https://mlsanity.com/paper/1704.02510/) (2017)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph](https://mlsanity.com/paper/2006.16934/) (2020)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [Multi-Content GAN for Few-Shot Font Style Transfer](https://mlsanity.com/paper/1712.00516/) (2017)
- [StructuralLM: Structural Pre-training for Form Understanding](https://mlsanity.com/paper/2105.11210/) (2021)
- [DG-Font: Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2104.03064/) (2021)
- [Separating Style and Content for Generalized Style Transfer](https://mlsanity.com/paper/1711.06454/) (2017)
- [A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation](https://mlsanity.com/paper/1809.01361/) (2018)
- [Artistic Glyph Image Synthesis via One-Stage Few-Shot Learning](https://mlsanity.com/paper/1910.04987/) (2019)
- [Few-shot Font Generation with Localized Style Representations and Factorization](https://mlsanity.com/paper/2009.11042/) (2020)
- [Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts](https://mlsanity.com/paper/2104.00887/) (2021)
- [Few-shot Compositional Font Generation with Dual Memory](https://mlsanity.com/paper/2005.10510/) (2020)
- [Learning to Write Stylized Chinese Characters by Reading a Handful of Examples](https://mlsanity.com/paper/1712.06424/) (2017)
- [CalliGAN: Style and Structure-aware Chinese Calligraphy Character Generator](https://mlsanity.com/paper/2005.12500/) (2020)

## Built on by (most cited first, 18 of 18)

- [FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning](https://mlsanity.com/paper/2312.12142/) (2023)
- [Disentangling Writer and Character Styles for Handwriting Generation](https://mlsanity.com/paper/2303.14736/) (2023)
- [CF-Font: Content Fusion for Few-shot Font Generation](https://mlsanity.com/paper/2303.14017/) (2023)
- [TextPainter: Multimodal Text Image Generation with Visual-harmony and Text-comprehension for Poster Design](https://mlsanity.com/paper/2308.04733/) (2023)
- [Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition](https://mlsanity.com/paper/2305.15660/) (2023)
- [Few shot font generation via transferring similarity guided global style and quantization local style](https://mlsanity.com/paper/2309.00827/) (2023)
- [Decoupling Layout from Glyph in Online Chinese Handwriting Generation](https://mlsanity.com/paper/2410.02309/) (2024)
- [FontTransformer: Few-shot High-resolution Chinese Glyph Image Synthesis via Stacked Transformers](https://mlsanity.com/paper/2210.06301/) (2022)
- [VQ-Font: Few-Shot Font Generation with Structure-Aware Enhancement and Quantization](https://mlsanity.com/paper/2308.14018/) (2023)
- [DeepCalliFont: Few-shot Chinese Calligraphy Font Synthesis by Integrating Dual-modality Generative Models](https://mlsanity.com/paper/2312.10314/) (2023)
- [Font Style Interpolation with Diffusion Models](https://mlsanity.com/paper/2402.14311/) (2024)
- [Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation](https://mlsanity.com/paper/2601.01593/) (2026)
- [FontCrafter: High-Fidelity Element-Driven Artistic Font Creation with Visual In-Context Generation](https://mlsanity.com/paper/2603.22054/) (2026)
- [Local Style Awareness of Font Images](https://mlsanity.com/paper/2310.06337/) (2023)
- [Total Disentanglement of Font Images into Style and Character Class Features](https://mlsanity.com/paper/2403.12784/) (2024)
- [Text Image Generation for Low-Resource Languages with Dual Translation Learning](https://mlsanity.com/paper/2409.17747/) (2024)
- [AnyArtisticGlyph: Multilingual Controllable Artistic Glyph Generation](https://mlsanity.com/paper/2504.04743/) (2025)
- [InkDiffuser: High-Fidelity One-shot Chinese Calligraphy via Differentiable Morphological Optimization](https://mlsanity.com/paper/2605.05865/) (2026)

## Abstract (arXiv)

Generating a new font library is a very labor-intensive and time-consuming job for glyph-rich scripts. Few-shot font generation is thus required, as it requires only a few glyph references without fine-tuning during test. Existing methods follow the style-content disentanglement paradigm and expect novel fonts to be produced by combining the style codes of the reference glyphs and the content representations of the source. However, these few-shot font generation methods either fail to capture content-independent style representations, or employ localized component-wise style representations, which is insufficient to model many Chinese font styles that involve hyper-component features such as inter-component spacing and "connected-stroke". To resolve these drawbacks and make the style representations more reliable, we propose a self-supervised cross-modality pre-training strategy and a cross-modality transformer-based encoder that is conditioned jointly on the glyph image and the corresponding stroke labels. The cross-modality encoder is pre-trained in a self-supervised manner to allow effective capture of cross- and intra-modality correlations, which facilitates the content-style disentanglement and modeling style representations of all scales (stroke-level, component-level and character-level). The pre-trained encoder is then applied to the downstream font generation task without fine-tuning. Experimental comparisons of our method with state-of-the-art methods demonstrate our method successfully transfers styles of all scales. In addition, it only requires one reference glyph and achieves the lowest rate of bad cases in the few-shot font generation task 28% lower than the second best