# Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models arXiv: https://arxiv.org/abs/2311.12796 · HTML: https://mlsanity.com/paper/2311.12796/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 53 references that are in the map. - First posted on arXiv 21 Nov 2023. ## Builds on (most cited first, 28 of 28) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020) - [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020) - [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022) - [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019) - [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023) - [3D Human Pose Estimation via Intuitive Physics](https://mlsanity.com/paper/2303.18246/) (2023) - [MultiScale MeshGraphNets](https://mlsanity.com/paper/2210.00612/) (2022) - [SNUG: Self-Supervised Neural Dynamic Garments](https://mlsanity.com/paper/2204.02219/) (2022) - [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020) - [Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On](https://mlsanity.com/paper/2105.06462/) (2021) - [HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics](https://mlsanity.com/paper/2212.07242/) (2022) - [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020) - [DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact](https://mlsanity.com/paper/2106.05306/) (2021) - [PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation](https://mlsanity.com/paper/2012.11310/) (2020) - [Neural Cloth Simulation](https://mlsanity.com/paper/2212.11220/) (2022) - [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022) - [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022) - [φ-SfT: Shape-from-Template with a Physics-Based Deformation Model](https://mlsanity.com/paper/2203.11938/) (2022) - [Neural Implicit Representations for Physical Parameter Inference from a Single Video](https://mlsanity.com/paper/2204.14030/) (2022) - [IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/1904.12144/) (2019) - [HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model](https://mlsanity.com/paper/1803.10193/) (2018) - [Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View](https://mlsanity.com/paper/1809.10305/) (2018) - [Fine-grained differentiable physics: a yarn-level model for fabrics](https://mlsanity.com/paper/2202.00504/) (2022) - [Learning Incompressible Fluid Dynamics from Scratch -- Towards Fast, Differentiable Fluid Models that Generalize](https://mlsanity.com/paper/2006.08762/) (2020) - [How Will It Drape Like? Capturing Fabric Mechanics from Depth Images](https://mlsanity.com/paper/2304.06704/) (2023) - [Hierarchical Cloth Simulation using Deep Neural Networks](https://mlsanity.com/paper/1802.03168/) (2018) - [Efficient Cloth Simulation using Miniature Cloth and Upscaling Deep Neural Networks](https://mlsanity.com/paper/1907.03953/) (2019) ## Built on by (most cited first, 8 of 8) - [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024) - [Metamizer: a versatile neural optimizer for fast and accurate physics simulations](https://mlsanity.com/paper/2410.19746/) (2024) - [Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields](https://mlsanity.com/paper/2503.19976/) (2025) - [Physics from Video: Identifiability of Time-Invariant Second-Order ODEs under Minimal Trajectory Conditions](https://mlsanity.com/paper/2606.00115/) (2026) - [Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints](https://mlsanity.com/paper/2507.22699/) (2025) - [SAFT: Shape and Appearance of Fabrics from Template via Differentiable Physical Simulations from Monocular Video](https://mlsanity.com/paper/2509.08828/) (2025) - [FNOPT: Resolution-Agnostic, Self-Supervised Cloth Simulation using Meta-Optimization with Fourier Neural Operators](https://mlsanity.com/paper/2512.05762/) (2025) - [Patch-based Representation and Learning for Efficient Deformation Modeling](https://mlsanity.com/paper/2601.05035/) (2026) ## Abstract (arXiv) 3D reconstruction of dynamic scenes is a long-standing problem in computer graphics and increasingly difficult the less information is available. Shape-from-Template (SfT) methods aim to reconstruct a template-based geometry from RGB images or video sequences, often leveraging just a single monocular camera without depth information, such as regular smartphone recordings. Unfortunately, existing reconstruction methods are either unphysical and noisy or slow in optimization. To solve this problem, we propose a novel SfT reconstruction algorithm for cloth using a pre-trained neural surrogate model that is fast to evaluate, stable, and produces smooth reconstructions due to a regularizing physics simulation. Differentiable rendering of the simulated mesh enables pixel-wise comparisons between the reconstruction and a target video sequence that can be used for a gradient-based optimization procedure to extract not only shape information but also physical parameters such as stretching, shearing, or bending stiffness of the cloth. This allows to retain a precise, stable, and smooth reconstructed geometry while reducing the runtime by a factor of 400-500 compared to φ-SfT, a state-of-the-art physics-based SfT approach.