# PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations

arXiv: https://arxiv.org/abs/2404.04421 · HTML: https://mlsanity.com/paper/2404.04421/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 86 of its 131 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 30 of 86)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [Effective Whole-body Pose Estimation with Two-stages Distillation](https://mlsanity.com/paper/2307.15880/) (2023)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [NeRF-Editing: Geometry Editing of Neural Radiance Fields](https://mlsanity.com/paper/2205.04978/) (2022)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians](https://mlsanity.com/paper/2312.02134/) (2023)

## Built on by (most cited first, 30 of 51)

- [Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior](https://mlsanity.com/paper/2503.01610/) (2025)
- [Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video](https://mlsanity.com/paper/2409.08189/) (2024)
- [Relightable Full-Body Gaussian Codec Avatars](https://mlsanity.com/paper/2501.14726/) (2025)
- [GauSTAR: Gaussian Surface Tracking and Reconstruction](https://mlsanity.com/paper/2501.10283/) (2025)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2407.15212/) (2024)
- [DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion](https://mlsanity.com/paper/2409.17145/) (2024)
- [Multimodal Latent Diffusion Model for Complex Sewing Pattern Generation](https://mlsanity.com/paper/2412.14453/) (2024)
- [TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting](https://mlsanity.com/paper/2503.17032/) (2025)
- [ToMiE: Towards Explicit Exoskeleton for the Reconstruction of Complicated 3D Human Avatars](https://mlsanity.com/paper/2410.08082/) (2024)
- [Sequential Gaussian Avatars with Hierarchical Motion Context](https://mlsanity.com/paper/2411.16768/) (2024)
- [PICA: Physics-Integrated Clothed Avatar](https://mlsanity.com/paper/2407.05324/) (2024)
- [MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics](https://mlsanity.com/paper/2510.01619/) (2025)
- [RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians](https://mlsanity.com/paper/2501.07104/) (2025)
- [GroomLight: Hybrid Inverse Rendering for Relightable Human Hair Appearance Modeling](https://mlsanity.com/paper/2503.10597/) (2025)
- [HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis](https://mlsanity.com/paper/2508.09137/) (2025)
- [MOSIV: Multi-Object System Identification from Videos](https://mlsanity.com/paper/2603.06022/) (2026)
- [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025)
- [PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics](https://mlsanity.com/paper/2410.07688/) (2024)
- [GGAvatar: Reconstructing Garment-Separated 3D Gaussian Splatting Avatars from Monocular Video](https://mlsanity.com/paper/2411.09952/) (2024)
- [ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation](https://mlsanity.com/paper/2506.05317/) (2025)
- [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025)
- [Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields](https://mlsanity.com/paper/2503.19976/) (2025)
- [UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment](https://mlsanity.com/paper/2506.01802/) (2025)
- [Parametric Gaussian Human Model: Generalizable Prior for Efficient and Realistic Human Avatar Modeling](https://mlsanity.com/paper/2506.06645/) (2025)
- [MonoCloth: Reconstruction and Animation of Cloth-Decoupled Human Avatars from Monocular Videos](https://mlsanity.com/paper/2508.04505/) (2025)
- [Image2Garment: Simulation-ready Garment Generation from a Single Image](https://mlsanity.com/paper/2601.09658/) (2026)
- [NeuROK: Generative 4D Neural Object Kinematics](https://mlsanity.com/paper/2605.30347/) (2026)
- [SVAD: From Single Image to 3D Avatar via Synthetic Data Generation with Video Diffusion and Data Augmentation](https://mlsanity.com/paper/2505.05475/) (2025)
- [SpringTime: Learning Simulatable Models of Cloth with Spatially-varying Constitutive Properties](https://mlsanity.com/paper/2507.21288/) (2025)

## Abstract (arXiv)

Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at: https://qingqing-zhao.github.io/PhysAvatar