# Textureless Deformable Surface Reconstruction with Invisible Markers

arXiv: https://arxiv.org/abs/2308.13678 · HTML: https://mlsanity.com/paper/2308.13678/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 72 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data](https://mlsanity.com/paper/2003.09572/) (2020)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wild](https://mlsanity.com/paper/2303.13652/) (2023)
- [Differentiable Event Stream Simulator for Non-Rigid 3D Tracking](https://mlsanity.com/paper/2104.15139/) (2021)
- [IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/1904.12144/) (2019)
- [Learning to Reconstruct Texture-less Deformable Surfaces from a Single View](https://mlsanity.com/paper/1803.08908/) (2018)
- [Enhancing Deformable Local Features by Jointly Learning to Detect and Describe Keypoints](https://mlsanity.com/paper/2304.00583/) (2023)
- [Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2302.02410/) (2023)
- [Better Together: Joint Reasoning for Non-rigid 3D Reconstruction with Specularities and Shading](https://mlsanity.com/paper/1708.01654/) (2017)
- [ROBUSfT: Robust Real-Time Shape-from-Template, a C++ Library](https://mlsanity.com/paper/2301.04037/) (2023)
- [A Dark Flash Normal Camera](https://mlsanity.com/paper/2012.06125/) (2020)
- [Simultaneous Surface Reflectance and Fluorescence Spectra Estimation](https://mlsanity.com/paper/1605.04243/) (2016)

## Abstract (arXiv)

Reconstructing and tracking deformable surface with little or no texture has posed long-standing challenges. Fundamentally, the challenges stem from textureless surfaces lacking features for establishing cross-image correspondences. In this work, we present a novel type of markers to proactively enrich the object's surface features, and thereby ease the 3D surface reconstruction and correspondence tracking. Our markers are made of fluorescent dyes, visible only under the ultraviolet (UV) light and invisible under regular lighting condition. Leveraging the markers, we design a multi-camera system that captures surface deformation under the UV light and the visible light in a time multiplexing fashion. Under the UV light, markers on the object emerge to enrich its surface texture, allowing high-quality 3D shape reconstruction and tracking. Under the visible light, markers become invisible, allowing us to capture the object's original untouched appearance. We perform experiments on various challenging scenes, including hand gestures, facial expressions, waving cloth, and hand-object interaction. In all these cases, we demonstrate that our system is able to produce robust, high-quality 3D reconstruction and tracking.