# Neural Non-Rigid Tracking

arXiv: https://arxiv.org/abs/2006.13240 · HTML: https://mlsanity.com/paper/2006.13240/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 30 of 49)

- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Monocular Dynamic View Synthesis: A Reality Check](https://mlsanity.com/paper/2210.13445/) (2022)
- [MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds](https://mlsanity.com/paper/2405.17421/) (2024)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects](https://mlsanity.com/paper/2303.14435/) (2023)
- [NPMs: Neural Parametric Models for 3D Deformable Shapes](https://mlsanity.com/paper/2104.00702/) (2021)
- [Lepard: Learning partial point cloud matching in rigid and deformable scenes](https://mlsanity.com/paper/2111.12591/) (2021)
- [Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction](https://mlsanity.com/paper/2312.01196/) (2023)
- [A Survey of Non-Rigid 3D Registration](https://mlsanity.com/paper/2203.07858/) (2022)
- [Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera](https://mlsanity.com/paper/2206.15258/) (2022)
- [Non-rigid Point Cloud Registration with Neural Deformation Pyramid](https://mlsanity.com/paper/2205.12796/) (2022)
- [DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation](https://mlsanity.com/paper/2312.00583/) (2023)
- [CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes](https://mlsanity.com/paper/2206.04382/) (2022)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction](https://mlsanity.com/paper/2203.07977/) (2022)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Event-based Non-Rigid Reconstruction from Contours](https://mlsanity.com/paper/2210.06270/) (2022)
- [GarmentTracking: Category-Level Garment Pose Tracking](https://mlsanity.com/paper/2303.13913/) (2023)
- [Deep Graph-based Spatial Consistency for Robust Non-rigid Point Cloud Registration](https://mlsanity.com/paper/2303.09950/) (2023)
- [Geometry in Style: 3D Stylization via Surface Normal Deformation](https://mlsanity.com/paper/2503.23241/) (2025)
- [PREF: Predictability Regularized Neural Motion Fields](https://mlsanity.com/paper/2209.10691/) (2022)
- [Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields](https://mlsanity.com/paper/2503.20776/) (2025)
- [Fast and Robust Non-Rigid Registration Using Accelerated Majorization-Minimization](https://mlsanity.com/paper/2206.03410/) (2022)
- [HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects](https://mlsanity.com/paper/2301.07652/) (2023)
- [Non-Rigid Shape Registration via Deep Functional Maps Prior](https://mlsanity.com/paper/2311.04494/) (2023)
- [SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation](https://mlsanity.com/paper/2409.11682/) (2024)
- [SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes](https://mlsanity.com/paper/2308.08258/) (2023)
- [DV-Matcher: Deformation-based Non-Rigid Point Cloud Matching Guided by Pre-trained Visual Features](https://mlsanity.com/paper/2408.08568/) (2024)
- [Neural Intrinsic Embedding for Non-rigid Point Cloud Matching](https://mlsanity.com/paper/2303.01038/) (2023)
- [Reconstruction of Manipulated Garment with Guided Deformation Prior](https://mlsanity.com/paper/2405.10934/) (2024)

## Abstract (arXiv)

We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization. Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences and their confidences. These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem. By enabling gradient back-propagation through the weighted non-linear least squares solver, we are able to learn correspondences and confidences in an end-to-end manner such that they are optimal for the task of non-rigid tracking. Under this formulation, correspondence confidences can be learned via self-supervision, informing a learned robust optimization, where outliers and wrong correspondences are automatically down-weighted to enable effective tracking. Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85 times faster correspondence prediction than comparable deep-learning based methods. We make our code available.