# DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion

arXiv: https://arxiv.org/abs/2303.09604 · HTML: https://mlsanity.com/paper/2303.09604/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Diffusion-GAN: Training GANs with Diffusion](https://mlsanity.com/paper/2206.02262/) (2022)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Multi-Content GAN for Few-Shot Font Style Transfer](https://mlsanity.com/paper/1712.00516/) (2017)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [Awesome Typography: Statistics-Based Text Effects Transfer](https://mlsanity.com/paper/1611.09026/) (2016)
- [Context-Aware Text-Based Binary Image Stylization and Synthesis](https://mlsanity.com/paper/1810.03767/) (2018)
- [Trick or TReAT: Thematic Reinforcement for Artistic Typography](https://mlsanity.com/paper/1903.07820/) (2019)

## Built on by (most cited first, 26 of 26)

- [Noise-Free Score Distillation](https://mlsanity.com/paper/2310.17590/) (2023)
- [COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design](https://mlsanity.com/paper/2311.16974/) (2023)
- [FonTS: Text Rendering with Typography and Style Controls](https://mlsanity.com/paper/2412.00136/) (2024)
- [AniClipart: Clipart Animation with Text-to-Video Priors](https://mlsanity.com/paper/2404.12347/) (2024)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [WordCon: Word-level Typography Control in Scene Text Rendering](https://mlsanity.com/paper/2506.21276/) (2025)
- [WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models](https://mlsanity.com/paper/2310.18332/) (2023)
- [Dynamic Typography: Bringing Text to Life via Video Diffusion Prior](https://mlsanity.com/paper/2404.11614/) (2024)
- [TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation](https://mlsanity.com/paper/2401.11094/) (2024)
- [FontStudio: Shape-Adaptive Diffusion Model for Coherent and Consistent Font Effect Generation](https://mlsanity.com/paper/2406.08392/) (2024)
- [Kinetic Typography Diffusion Model](https://mlsanity.com/paper/2407.10476/) (2024)
- [Intelligent Artistic Typography: A Comprehensive Review of Artistic Text Design and Generation](https://mlsanity.com/paper/2407.14774/) (2024)
- [OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic Typography](https://mlsanity.com/paper/2506.21101/) (2025)
- [MetaDesigner: Advancing Artistic Typography Through AI-Driven, User-Centric, and Multilingual WordArt Synthesis](https://mlsanity.com/paper/2406.19859/) (2024)
- [Typographic Text Generation with Off-the-Shelf Diffusion Model](https://mlsanity.com/paper/2402.14314/) (2024)
- [GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models](https://mlsanity.com/paper/2408.07259/) (2024)
- [VitaGlyph: Vitalizing Artistic Typography with Flexible Dual-branch Diffusion Models](https://mlsanity.com/paper/2410.01738/) (2024)
- [Font Style Interpolation with Diffusion Models](https://mlsanity.com/paper/2402.14311/) (2024)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [Khattat: Enhancing Readability and Concept Representation of Semantic Typography](https://mlsanity.com/paper/2410.03748/) (2024)
- [Towards Visual Text Design Transfer Across Languages](https://mlsanity.com/paper/2410.18823/) (2024)
- [Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models](https://mlsanity.com/paper/2608.29137/) (2026)
- [Cross-Domain Image Conversion by CycleDM](https://mlsanity.com/paper/2403.02919/) (2024)
- [WordCraft: Interactive Artistic Typography with Attention Awareness and Noise Blending](https://mlsanity.com/paper/2507.09573/) (2025)
- [FontFusion: Enhancing Generative Text in Diffusion Models with Typographic Conditioning](https://mlsanity.com/paper/2606.06066/) (2026)
- [Strike a Chord! Modal Kinetic Typography](https://mlsanity.com/paper/2609.38325/) (2026)

## Abstract (arXiv)

We introduce a novel method to automatically generate an artistic typography by stylizing one or more letter fonts to visually convey the semantics of an input word, while ensuring that the output remains readable. To address an assortment of challenges with our task at hand including conflicting goals (artistic stylization vs. legibility), lack of ground truth, and immense search space, our approach utilizes large language models to bridge texts and visual images for stylization and build an unsupervised generative model with a diffusion model backbone. Specifically, we employ the denoising generator in Latent Diffusion Model (LDM), with the key addition of a CNN-based discriminator to adapt the input style onto the input text. The discriminator uses rasterized images of a given letter/word font as real samples and output of the denoising generator as fake samples. Our model is coined DS-Fusion for discriminated and stylized diffusion. We showcase the quality and versatility of our method through numerous examples, qualitative and quantitative evaluation, as well as ablation studies. User studies comparing to strong baselines including CLIPDraw and DALL-E 2, as well as artist-crafted typographies, demonstrate strong performance of DS-Fusion.