mlsanityData to 14 Oct 2026

Synthesizing Environment-Specific People in Photographs

arXiv 2312.14579 ↗First posted 22 Dec 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We present ESP, a novel method for context-aware full-body generation, that enables photo-realistic synthesis and inpainting of people wearing clothing that is semantically appropriate for the scene depicted in an input photograph. ESP is conditioned on a 2D pose and contextual cues that are extracted from the photograph of the scene and integrated into the generation process, where the clothing is modeled explicitly with human parsing masks (HPM). Generated HPMs are used as tight guiding masks for inpainting, such that no changes are made to the original background. Our models are trained on a dataset containing a set of in-the-wild photographs of people covering a wide range of different environments. The method is analyzed quantitatively and qualitatively, and we show that ESP outperforms the state-of-the-art on the task of contextual full-body generation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

45 of 58 references.

References · 45

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
33 more
  1. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  6. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  7. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  8. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  9. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  10. 2021Advances in Neural Renderingcited by 365
  11. 2020UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentationcited by 332
  12. 2018Everybody Dance Nowcited by 315
  13. 2020State of the Art on Neural Renderingcited by 278
  14. 2017Pose Guided Person Image Generationcited by 251
  15. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 220
  16. 2019Pluralistic Image Completioncited by 162
  17. 2020Controllable Person Image Synthesis with Attribute-Decomposed GANcited by 150
  18. 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 144
  19. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  20. 2020Deep Image Spatial Transformation for Person Image Generationcited by 100
  21. 2016PixelVAE: A Latent Variable Model for Natural Imagescited by 70
  22. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  23. 2019Boundless: Generative Adversarial Networks for Image Extensioncited by 62
  24. 2021StylePeople: A Generative Model of Fullbody Human Avatarscited by 62
  25. 2021Style and Pose Control for Image Synthesis of Humans from a Single Monocular Viewcited by 53
  26. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  27. 2021HumanGAN: A Generative Model of Humans Imagescited by 47
  28. 2022InsetGAN for Full-Body Image Generationcited by 45
  29. 2021Human De-occlusion: Invisible Perception and Recovery for Humanscited by 31
  30. 2019Appearance and Pose-Conditioned Human Image Generation using Deformable GANscited by 24
  31. 2021Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistencycited by 15
  32. 2021Hallucinating Pose-Compatible Scenescited by 14
  33. 2019Controllable and Progressive Image Extrapolationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordancecited by 61
  2. 2025Insert Anything: Image Insertion via In-Context Editing in DiTcited by 61
  3. 2026MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animationcited by 3
  4. 2024Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimatorscited by 1