# Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation

arXiv: https://arxiv.org/abs/2412.15853 · HTML: https://mlsanity.com/paper/2412.15853/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 11 of its 88 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 11 of 11)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [TextStyleBrush: Transfer of Text Aesthetics from a Single Example](https://mlsanity.com/paper/2106.08385/) (2021)
- [Diffusion models for Handwriting Generation](https://mlsanity.com/paper/2011.06704/) (2020)
- [VATr++: Choose Your Words Wisely for Handwritten Text Generation](https://mlsanity.com/paper/2402.10798/) (2024)
- [Generative Adversarial Network for Handwritten Text](https://mlsanity.com/paper/1907.11845/) (2019)
- [Generating Synthetic Data for Text Recognition](https://mlsanity.com/paper/1608.04224/) (2016)
- [Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models](https://mlsanity.com/paper/2409.00786/) (2024)

## Built on by (most cited first, 2 of 2)

- [Autoregressive Styled Text Image Generation, but Make it Reliable](https://mlsanity.com/paper/2510.23240/) (2025)
- [Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation](https://mlsanity.com/paper/2508.17017/) (2025)

## Abstract (arXiv)

The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.