# Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models

arXiv: https://arxiv.org/abs/2409.00786 · HTML: https://mlsanity.com/paper/2409.00786/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [ScrabbleGAN: Semi-Supervised Varying Length Handwritten Text Generation](https://mlsanity.com/paper/2003.10557/) (2020)
- [GANwriting: Content-Conditioned Generation of Styled Handwritten Word Images](https://mlsanity.com/paper/2003.02567/) (2020)
- [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://mlsanity.com/paper/1910.04396/) (2019)
- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [DeepWriting: Making Digital Ink Editable via Deep Generative Modeling](https://mlsanity.com/paper/1801.08379/) (2018)
- [Handwriting Transformers](https://mlsanity.com/paper/2104.03964/) (2021)
- [Text and Style Conditioned GAN for Generation of Offline Handwriting Lines](https://mlsanity.com/paper/2009.00678/) (2020)
- [Content and Style Aware Generation of Text-line Images for Handwriting Recognition](https://mlsanity.com/paper/2204.05539/) (2022)
- [Disentangling Writer and Character Styles for Handwriting Generation](https://mlsanity.com/paper/2303.14736/) (2023)
- [Few-Shot Font Generation by Learning Fine-Grained Local Styles](https://mlsanity.com/paper/2205.09965/) (2022)
- [Handwritten Text Generation from Visual Archetypes](https://mlsanity.com/paper/2303.15269/) (2023)
- [SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Text](https://mlsanity.com/paper/2202.11456/) (2022)
- [Diffusion models for Handwriting Generation](https://mlsanity.com/paper/2011.06704/) (2020)

## Built on by (most cited first, 5 of 5)

- [Zero-Shot Styled Text Image Generation, but Make It Autoregressive](https://mlsanity.com/paper/2503.17074/) (2025)
- [Autoregressive Styled Text Image Generation, but Make it Reliable](https://mlsanity.com/paper/2510.23240/) (2025)
- [Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation](https://mlsanity.com/paper/2412.15853/) (2024)
- [Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation](https://mlsanity.com/paper/2508.17017/) (2025)
- [Different Strokes for Different Folks: Writer Identification for Historical Arabic Manuscripts](https://mlsanity.com/paper/2604.22515/) (2026)

## Abstract (arXiv)

The imitation of cursive handwriting is mainly limited to generating handwritten words or lines. Multiple synthetic outputs must be stitched together to create paragraphs or whole pages, whereby consistency and layout information are lost. To close this gap, we propose a method for imitating handwriting at the paragraph level that also works for unseen writing styles. Therefore, we introduce a modified latent diffusion model that enriches the encoder-decoder mechanism with specialized loss functions that explicitly preserve the style and content. We enhance the attention mechanism of the diffusion model with adaptive 2D positional encoding and the conditioning mechanism to work with two modalities simultaneously: a style image and the target text. This significantly improves the realism of the generated handwriting. We set a new benchmark in our comprehensive evaluation, achieving 61 % mAP and 56 % top-1 accuracy in style preservation, significantly outperforming the previous best method (37 % mAP, 30 % top-1). We are making our code publicly available for reproducibility, supporting research in this area and research into potential countermeasures: https://github.com/M4rt1nM4yr/paragraph_handwriting_imitation_ldm