# PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing arXiv: https://arxiv.org/abs/2303.00137 ยท HTML: https://mlsanity.com/paper/2303.00137/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 74 references that are in the map. - First posted on arXiv 28 Feb 2023. ## Builds on (most cited first, 30 of 32) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015) - [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017) - [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019) - [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016) - [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020) - [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020) - [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017) - [DeepHuman: 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/1903.06473/) (2019) - [Monocular Total Capture: Posing Face, Body, and Hands in the Wild](https://mlsanity.com/paper/1812.01598/) (2018) - [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019) - [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019) - [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017) - [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017) - [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021) - [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019) - [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019) - [Region-aware Adaptive Instance Normalization for Image Harmonization](https://mlsanity.com/paper/2106.02853/) (2021) - [SSH: A Self-Supervised Framework for Image Harmonization](https://mlsanity.com/paper/2108.06805/) (2021) - [Foreground-aware Semantic Representations for Image Harmonization](https://mlsanity.com/paper/2006.00809/) (2020) - [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021) - [SurfNet: Generating 3D shape surfaces using deep residual networks](https://mlsanity.com/paper/1703.04079/) (2017) - [Image2Mesh: A Learning Framework for Single Image 3D Reconstruction](https://mlsanity.com/paper/1711.10669/) (2017) - [SSN: Soft Shadow Network for Image Compositing](https://mlsanity.com/paper/2007.08211/) (2020) - [Controllable Shadow Generation Using Pixel Height Maps](https://mlsanity.com/paper/2207.05385/) (2022) - [People as Scene Probes](https://mlsanity.com/paper/2007.09209/) (2020) - [Design and Deployment of Photo2Building: A Cloud-based Procedural Modeling Tool as a Service](https://mlsanity.com/paper/2008.01286/) (2020) ## Built on by (most cited first, 15 of 15) - [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023) - [IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation](https://mlsanity.com/paper/2403.10701/) (2024) - [Shadow Generation for Composite Image Using Diffusion model](https://mlsanity.com/paper/2403.15234/) (2024) - [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024) - [GlobalMapper: Arbitrary-Shaped Urban Layout Generation](https://mlsanity.com/paper/2307.09693/) (2023) - [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024) - [Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment](https://mlsanity.com/paper/2412.00306/) (2024) - [MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis](https://mlsanity.com/paper/2412.02635/) (2024) - [Floating No More: Object-Ground Reconstruction from a Single Image](https://mlsanity.com/paper/2407.18914/) (2024) - [All-frequency Full-body Human Image Relighting](https://mlsanity.com/paper/2411.00356/) (2024) - [Refracting Reality: Generating Images with Realistic Transparent Objects](https://mlsanity.com/paper/2511.17340/) (2025) - [Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation](https://mlsanity.com/paper/2511.22184/) (2025) - [Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps](https://mlsanity.com/paper/2602.21820/) (2026) - [MultiShadow: Multi-Object Shadow Generation for Image Compositing via Diffusion Model](https://mlsanity.com/paper/2603.02743/) (2026) - [VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion](https://mlsanity.com/paper/2603.08020/) (2026) ## Abstract (arXiv) Lighting effects such as shadows or reflections are key in making synthetic images realistic and visually appealing. To generate such effects, traditional computer graphics uses a physically-based renderer along with 3D geometry. To compensate for the lack of geometry in 2D Image compositing, recent deep learning-based approaches introduced a pixel height representation to generate soft shadows and reflections. However, the lack of geometry limits the quality of the generated soft shadows and constrain reflections to pure specular ones. We introduce PixHt-Lab, a system leveraging an explicit mapping from pixel height representation to 3D space. Using this mapping, PixHt-Lab reconstructs both the cutout and background geometry and renders realistic, diverse, lighting effects for image compositing. Given a surface with physically-based materials, we can render reflections with varying glossiness. To generate more realistic soft shadows, we further propose to use 3D-aware buffer channels to guide a neural renderer. Both quantitative and qualitative evaluations demonstrate that PixHt-Lab significantly improves soft shadow generation.