mlsanityData to 7 Oct 2026

KPE: Keypoint Pose Encoding for Transformer-based Image Generation

arXiv 2203.04907 ↗First posted 9 Mar 2022cs.CV cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Transformers have recently been shown to generate high quality images from text input. However, the existing method of pose conditioning using skeleton image tokens is computationally inefficient and generate low quality images. Therefore we propose a new method; Keypoint Pose Encoding (KPE); KPE is 10 times more memory efficient and over 73% faster at generating high quality images from text input conditioned on the pose. The pose constraint improves the image quality and reduces errors on body extremities such as arms and legs. The additional benefits include invariance to changes in the target image domain and image resolution, making it easily scalable to higher resolution images. We demonstrate the versatility of KPE by generating photorealistic multiperson images derived from the DeepFashion dataset. We also introduce a evaluation method People Count Error (PCE) that is effective in detecting error in generated human images.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 9 Mar 2022.

Builds on

34 of 43 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
22 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2014Conditional Generative Adversarial Netscited by 2,152
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  6. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  7. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2019Axial Attention in Multidimensional Transformerscited by 350
  10. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  11. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  12. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  13. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  14. 2017Pose Guided Person Image Generationcited by 251
  15. 2018A Note on the Inception Scorecited by 225
  16. 2017Deformable GANs for Pose-based Human Image Generationcited by 166
  17. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  18. 2019Progressive Pose Attention Transfer for Person Image Generationcited by 138
  19. 2014Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimationcited by 129
  20. 2021Pose Recognition with Cascade Transformerscited by 99
  21. 2021Skeletor: Skeletal Transformers for Robust Body-Pose Estimationcited by 26
  22. 2020Towards Fine-grained Human Pose Transfer with Detail Replenishing Networkcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 2; 2026: 1.

12 papers.

  1. 2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 104
  2. 2023SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Imagecited by 30
  3. 2024Boosting Camera Motion Control for Video Diffusion Transformerscited by 20
  4. 2024GRPose: Learning Graph Relations for Human Image Generation with Pose Priorscited by 14
  5. 2023UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfercited by 12
  6. 2024Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generationcited by 12
  7. 2024BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusioncited by 9
  8. 2023ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNetcited by 7
  9. 2023FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injectioncited by 6
  10. 2025Enhanced Multi-Scale Cross-Attention for Person Image Generationcited by 3
  11. 2025KB-DMGen: Knowledge-Based Global Guidance and Dynamic Pose Masking for Human Image Generationcited by 1
  12. 2026TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generationcited by 1