# Scene-aware Human Pose Generation using Transformer

arXiv: https://arxiv.org/abs/2308.02177 · HTML: https://mlsanity.com/paper/2308.02177/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Compositional Human Pose Regression](https://mlsanity.com/paper/1704.00159/) (2017)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [Pose Recognition with Cascade Transformers](https://mlsanity.com/paper/2104.06976/) (2021)
- [ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing](https://mlsanity.com/paper/1803.01837/) (2018)
- [The Pose Knows: Video Forecasting by Generating Pose Futures](https://mlsanity.com/paper/1705.00053/) (2017)
- [Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments](https://mlsanity.com/paper/1903.05690/) (2019)
- [Multi-Context Attention for Human Pose Estimation](https://mlsanity.com/paper/1702.07432/) (2017)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars](https://mlsanity.com/paper/2205.08535/) (2022)
- [Human Pose Regression by Combining Indirect Part Detection and Contextual Information](https://mlsanity.com/paper/1710.02322/) (2017)
- [Learning to Generate Synthetic Data via Compositing](https://mlsanity.com/paper/1904.05475/) (2019)
- [UniPose: Unified Human Pose Estimation in Single Images and Videos](https://mlsanity.com/paper/2001.08095/) (2020)
- [Binge Watching: Scaling Affordance Learning from Sitcoms](https://mlsanity.com/paper/1804.03080/) (2018)
- [Context-Aware Synthesis and Placement of Object Instances](https://mlsanity.com/paper/1812.02350/) (2018)
- [Multi-Person Pose Estimation with Enhanced Channel-wise and Spatial Information](https://mlsanity.com/paper/1905.03466/) (2019)
- [OPA: Object Placement Assessment Dataset](https://mlsanity.com/paper/2107.01889/) (2021)

## Built on by (most cited first, 9 of 9)

- [Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing](https://mlsanity.com/paper/2405.04496/) (2024)
- [Teleportraits: Training-Free People Insertion into Any Scene](https://mlsanity.com/paper/2510.05660/) (2025)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [Exploring Mutual Cross-Modal Attention for Context-Aware Human Affordance Generation](https://mlsanity.com/paper/2502.13637/) (2025)
- [HOIverse: A Synthetic Scene Graph Dataset With Human Object Interactions](https://mlsanity.com/paper/2506.19639/) (2025)
- [Deeply Dual Supervised learning for melanoma recognition](https://mlsanity.com/paper/2508.01994/) (2025)
- [Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsampling](https://mlsanity.com/paper/2508.06805/) (2025)
- [DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentation](https://mlsanity.com/paper/2508.06816/) (2025)
- [VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagation](https://mlsanity.com/paper/2508.06819/) (2025)

## Abstract (arXiv)

Affordance learning considers the interaction opportunities for an actor in the scene and thus has wide application in scene understanding and intelligent robotics. In this paper, we focus on contextual affordance learning, i.e., using affordance as context to generate a reasonable human pose in a scene. Existing scene-aware human pose generation methods could be divided into two categories depending on whether using pose templates. Our proposed method belongs to the template-based category, which benefits from the representative pose templates. Moreover, inspired by recent transformer-based methods, we associate each query embedding with a pose template, and use the interaction between query embeddings and scene feature map to effectively predict the scale and offsets for each pose template. In addition, we employ knowledge distillation to facilitate the offset learning given the predicted scale. Comprehensive experiments on Sitcom dataset demonstrate the effectiveness of our method.