# Kinetic Typography Diffusion Model

arXiv: https://arxiv.org/abs/2407.10476 · HTML: https://mlsanity.com/paper/2407.10476/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 54 of its 77 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](https://mlsanity.com/paper/2305.10474/) (2023)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023)
- [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023)
- [GlyphControl: Glyph Conditional Control for Visual Text Generation](https://mlsanity.com/paper/2305.18259/) (2023)
- [Multi-Content GAN for Few-Shot Font Style Transfer](https://mlsanity.com/paper/1712.00516/) (2017)
- [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://mlsanity.com/paper/2112.02815/) (2021)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [SwapText: Image Based Texts Transfer in Scenes](https://mlsanity.com/paper/2003.08152/) (2020)
- [TextStyleBrush: Transfer of Text Aesthetics from a Single Example](https://mlsanity.com/paper/2106.08385/) (2021)
- [FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning](https://mlsanity.com/paper/2312.12142/) (2023)
- [Controllable Artistic Text Style Transfer via Shape-Matching GAN](https://mlsanity.com/paper/1905.01354/) (2019)
- [Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach](https://mlsanity.com/paper/2011.14021/) (2020)
- [Unifying Multimodal Transformer for Bi-directional Image and Text Generation](https://mlsanity.com/paper/2110.09753/) (2021)

## Built on by (most cited first, 9 of 9)

- [BannerAgency: Advertising Banner Design with Multimodal LLM Agents](https://mlsanity.com/paper/2503.11060/) (2025)
- [Video Color Grading via Look-Up Table Generation](https://mlsanity.com/paper/2508.00548/) (2025)
- [Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models](https://mlsanity.com/paper/2605.19398/) (2026)
- [TriMotion: Modality-Agnostic Camera Control for Video Generation](https://mlsanity.com/paper/2606.20774/) (2026)
- [TransText: Alpha-as-RGB Representation for Transparent Text Animation](https://mlsanity.com/paper/2603.17944/) (2026)
- [VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation](https://mlsanity.com/paper/2610.01499/) (2026)
- [MG-Gen: Single Image to Motion Graphics Generation](https://mlsanity.com/paper/2504.02361/) (2025)
- [Bridging Online and Offline Handwriting via Differentiable Physical Rendering](https://mlsanity.com/paper/2608.03198/) (2026)
- [Strike a Chord! Modal Kinetic Typography](https://mlsanity.com/paper/2609.38325/) (2026)

## Abstract (arXiv)

This paper introduces a method for realistic kinetic typography that generates user-preferred animatable 'text content'. We draw on recent advances in guided video diffusion models to achieve visually-pleasing text appearances. To do this, we first construct a kinetic typography dataset, comprising about 600K videos. Our dataset is made from a variety of combinations in 584 templates designed by professional motion graphics designers and involves changing each letter's position, glyph, and size (i.e., flying, glitches, chromatic aberration, reflecting effects, etc.). Next, we propose a video diffusion model for kinetic typography. For this, there are three requirements: aesthetic appearances, motion effects, and readable letters. This paper identifies the requirements. For this, we present static and dynamic captions used as spatial and temporal guidance of a video diffusion model, respectively. The static caption describes the overall appearance of the video, such as colors, texture and glyph which represent a shape of each letter. The dynamic caption accounts for the movements of letters and backgrounds. We add one more guidance with zero convolution to determine which text content should be visible in the video. We apply the zero convolution to the text content, and impose it on the diffusion model. Lastly, our glyph loss, only minimizing a difference between the predicted word and its ground-truth, is proposed to make the prediction letters readable. Experiments show that our model generates kinetic typography videos with legible and artistic letter motions based on text prompts.