# A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation

arXiv: https://arxiv.org/abs/1602.03725 · HTML: https://mlsanity.com/paper/1602.03725/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2016.

## Built on by (most cited first, 30 of 42)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [GauHuman: Articulated Gaussian Splatting from Monocular Human Videos](https://mlsanity.com/paper/2312.02973/) (2023)
- [GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis](https://mlsanity.com/paper/2312.11458/) (2023)
- [Gaussian Shell Maps for Efficient 3D Human Generation](https://mlsanity.com/paper/2311.17857/) (2023)
- [LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors](https://mlsanity.com/paper/2205.15410/) (2022)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Differentiable Surface Rendering via Non-Differentiable Sampling](https://mlsanity.com/paper/2108.04886/) (2021)
- [3D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.10123/) (2023)
- [Differentiable Rendering of Neural SDFs through Reparameterization](https://mlsanity.com/paper/2206.05344/) (2022)
- [LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer](https://mlsanity.com/paper/2405.07319/) (2024)
- [LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment](https://mlsanity.com/paper/2402.17171/) (2024)
- [PAPR: Proximity Attention Point Rendering](https://mlsanity.com/paper/2307.11086/) (2023)
- [Gaussian Shadow Casting for Neural Characters](https://mlsanity.com/paper/2401.06116/) (2024)
- [Rasterized Edge Gradients: Handling Discontinuities Differentiably](https://mlsanity.com/paper/2405.02508/) (2024)
- [When Gaussian Meets Surfel: Ultra-fast High-fidelity Radiance Field Rendering](https://mlsanity.com/paper/2504.17545/) (2025)
- [GenDR: A Generalized Differentiable Renderer](https://mlsanity.com/paper/2204.13845/) (2022)
- [Efficient and Differentiable Shadow Computation for Inverse Problems](https://mlsanity.com/paper/2104.00359/) (2021)
- [The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility](https://mlsanity.com/paper/2311.16937/) (2023)
- [Plateau-reduced Differentiable Path Tracing](https://mlsanity.com/paper/2211.17263/) (2022)
- [ADOP: Approximate Differentiable One-Pixel Point Rendering](https://mlsanity.com/paper/2110.06635/) (2021)
- [Inverse Global Illumination using a Neural Radiometric Prior](https://mlsanity.com/paper/2305.02192/) (2023)
- [LiveHPS++: Robust and Coherent Motion Capture in Dynamic Free Environment](https://mlsanity.com/paper/2407.09833/) (2024)
- [GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes](https://mlsanity.com/paper/2106.13215/) (2021)
- [Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics](https://mlsanity.com/paper/2308.05739/) (2023)
- [Differentiable Diffusion for Dense Depth Estimation from Multi-view Images](https://mlsanity.com/paper/2106.08917/) (2021)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [Deep Portrait Lighting Enhancement with 3D Guidance](https://mlsanity.com/paper/2108.02121/) (2021)
- [FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics](https://mlsanity.com/paper/2111.00110/) (2021)
- [Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization](https://mlsanity.com/paper/2204.01386/) (2022)

## Abstract (arXiv)

Generative reconstruction methods compute the 3D configuration (such as pose and/or geometry) of a shape by optimizing the overlap of the projected 3D shape model with images. Proper handling of occlusions is a big challenge, since the visibility function that indicates if a surface point is seen from a camera can often not be formulated in closed form, and is in general discrete and non-differentiable at occlusion boundaries. We present a new scene representation that enables an analytically differentiable closed-form formulation of surface visibility. In contrast to previous methods, this yields smooth, analytically differentiable, and efficient to optimize pose similarity energies with rigorous occlusion handling, fewer local minima, and experimentally verified improved convergence of numerical optimization. The underlying idea is a new image formation model that represents opaque objects by a translucent medium with a smooth Gaussian density distribution which turns visibility into a smooth phenomenon. We demonstrate the advantages of our versatile scene model in several generative pose estimation problems, namely marker-less multi-object pose estimation, marker-less human motion capture with few cameras, and image-based 3D geometry estimation.