# Shape-from-Template with Generalised Camera

arXiv: https://arxiv.org/abs/2508.13791 · HTML: https://mlsanity.com/paper/2508.13791/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 127 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [Motion Guided 3D Pose Estimation from Videos](https://mlsanity.com/paper/2004.13985/) (2020)
- [XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera](https://mlsanity.com/paper/1907.00837/) (2019)
- [AnimalTrack: A Benchmark for Multi-Animal Tracking in the Wild](https://mlsanity.com/paper/2205.00158/) (2022)
- [State of the Art in Dense Monocular Non-Rigid 3D Reconstruction](https://mlsanity.com/paper/2210.15664/) (2022)
- [Survey on 3D face reconstruction from uncalibrated images](https://mlsanity.com/paper/2011.05740/) (2020)
- [Perspective Projection-Based 3D CT Reconstruction from Biplanar X-rays](https://mlsanity.com/paper/2303.05297/) (2023)
- [Decaf: Monocular Deformation Capture for Face and Hand Interactions](https://mlsanity.com/paper/2309.16670/) (2023)
- [Global registration of multiple point clouds using semidefinite programming](https://mlsanity.com/paper/1306.5226/) (2013)
- [A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data](https://mlsanity.com/paper/1511.02746/) (2015)
- [Organic Priors in Non-Rigid Structure from Motion](https://mlsanity.com/paper/2207.06262/) (2022)
- [Efficient Relative Pose Estimation for Cameras and Generalized Cameras in Case of Known Relative Rotation Angle](https://mlsanity.com/paper/1901.11357/) (2019)
- [ROBUSfT: Robust Real-Time Shape-from-Template, a C++ Library](https://mlsanity.com/paper/2301.04037/) (2023)
- [Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery](https://mlsanity.com/paper/2204.01276/) (2022)
- [Concise and Effective Network for 3D Human Modeling from Orthogonal Silhouettes](https://mlsanity.com/paper/1912.11616/) (2019)
- [Animals in Virtual Environments](https://mlsanity.com/paper/1912.12763/) (2019)
- [Design and Evaluation of a Generic Visual SLAM Framework for Multi-Camera Systems](https://mlsanity.com/paper/2210.07315/) (2022)
- [Procrustes Analysis with Deformations: A Closed-Form Solution by Eigenvalue Decomposition](https://mlsanity.com/paper/2206.14528/) (2022)
- [Automatic 3D Registration of Dental CBCT and Face Scan Data using 2D Projection Images](https://mlsanity.com/paper/2305.10132/) (2023)

## Built on by (most cited first, 1 of 1)

- [Globally Optimal Pose from Orthographic Silhouettes](https://mlsanity.com/paper/2604.09199/) (2026)

## Abstract (arXiv)

This article presents a new method for non-rigidly registering a 3D shape to 2D keypoints observed by a constellation of multiple cameras. Non-rigid registration of a 3D shape to observed 2D keypoints, i.e., Shape-from-Template (SfT), has been widely studied using single images, but SfT with information from multiple-cameras jointly opens new directions for extending the scope of known use-cases such as 3D shape registration in medical imaging and registration from hand-held cameras, to name a few. We represent such multi-camera setup with the generalised camera model; therefore any collection of perspective or orthographic cameras observing any deforming object can be registered. We propose multiple approaches for such SfT: the first approach where the corresponded keypoints lie on a direction vector from a known 3D point in space, the second approach where the corresponded keypoints lie on a direction vector from an unknown 3D point in space but with known orientation w.r.t some local reference frame, and a third approach where, apart from correspondences, the silhouette of the imaged object is also known. Together, these form the first set of solutions to the SfT problem with generalised cameras. The key idea behind SfT with generalised camera is the improved reconstruction accuracy from estimating deformed shape while utilising the additional information from the mutual constraints between multiple views of a deformed object. The correspondence-based approaches are solved with convex programming while the silhouette-based approach is an iterative refinement of the results from the convex solutions. We demonstrate the accuracy of our proposed methods on many synthetic and real data