# NRST: Non-rigid Surface Tracking from Monocular Video

arXiv: https://arxiv.org/abs/2107.02407 · HTML: https://mlsanity.com/paper/2107.02407/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 46 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 3 of 3)

- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [MonoPerfCap: Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1708.02136/) (2017)
- [Better Together: Joint Reasoning for Non-rigid 3D Reconstruction with Specularities and Shading](https://mlsanity.com/paper/1708.01654/) (2017)

## Built on by (most cited first, 5 of 5)

- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model](https://mlsanity.com/paper/2206.08368/) (2022)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields](https://mlsanity.com/paper/2307.12909/) (2023)
- [Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fields](https://mlsanity.com/paper/2503.19976/) (2025)

## Abstract (arXiv)

We propose an efficient method for non-rigid surface tracking from monocular RGB videos. Given a video and a template mesh, our algorithm sequentially registers the template non-rigidly to each frame. We formulate the per-frame registration as an optimization problem that includes a novel texture term specifically tailored towards tracking objects with uniform texture but fine-scale structure, such as the regular micro-structural patterns of fabric. Our texture term exploits the orientation information in the micro-structures of the objects, e.g., the yarn patterns of fabrics. This enables us to accurately track uniformly colored materials that have these high frequency micro-structures, for which traditional photometric terms are usually less effective. The results demonstrate the effectiveness of our method on both general textured non-rigid objects and monochromatic fabrics.