# KPE: Keypoint Pose Encoding for Transformer-based Image Generation

arXiv: https://arxiv.org/abs/2203.04907 · HTML: https://mlsanity.com/paper/2203.04907/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Deformable GANs for Pose-based Human Image Generation](https://mlsanity.com/paper/1801.00055/) (2017)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [Progressive Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/1904.03349/) (2019)

## Built on by (most cited first, 12 of 12)

- [HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation](https://mlsanity.com/paper/2304.04269/) (2023)
- [SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image](https://mlsanity.com/paper/2309.01855/) (2023)
- [Boosting Camera Motion Control for Video Diffusion Transformers](https://mlsanity.com/paper/2410.10802/) (2024)
- [GRPose: Learning Graph Relations for Human Image Generation with Pose Priors](https://mlsanity.com/paper/2408.16540/) (2024)
- [UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfer](https://mlsanity.com/paper/2304.08870/) (2023)
- [Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation](https://mlsanity.com/paper/2406.02485/) (2024)
- [BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion](https://mlsanity.com/paper/2404.04544/) (2024)
- [ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet](https://mlsanity.com/paper/2312.03154/) (2023)
- [FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection](https://mlsanity.com/paper/2312.09252/) (2023)
- [Enhanced Multi-Scale Cross-Attention for Person Image Generation](https://mlsanity.com/paper/2501.08900/) (2025)
- [KB-DMGen: Knowledge-Based Global Guidance and Dynamic Pose Masking for Human Image Generation](https://mlsanity.com/paper/2507.20083/) (2025)
- [TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation](https://mlsanity.com/paper/2606.07053/) (2026)

## Abstract (arXiv)

Transformers have recently been shown to generate high quality images from text input. However, the existing method of pose conditioning using skeleton image tokens is computationally inefficient and generate low quality images. Therefore we propose a new method; Keypoint Pose Encoding (KPE); KPE is 10 times more memory efficient and over 73% faster at generating high quality images from text input conditioned on the pose. The pose constraint improves the image quality and reduces errors on body extremities such as arms and legs. The additional benefits include invariance to changes in the target image domain and image resolution, making it easily scalable to higher resolution images. We demonstrate the versatility of KPE by generating photorealistic multiperson images derived from the DeepFashion dataset. We also introduce a evaluation method People Count Error (PCE) that is effective in detecting error in generated human images.