# Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a Single Image

arXiv: https://arxiv.org/abs/2210.15200 · HTML: https://mlsanity.com/paper/2210.15200/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 54 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Fitting a 3D Morphable Model to Edges: A Comparison Between Hard and Soft Correspondences](https://mlsanity.com/paper/1602.01125/) (2016)
- [Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review](https://mlsanity.com/paper/1905.02845/) (2019)
- [Landmark Detection and 3D Face Reconstruction for Caricature using a Nonlinear Parametric Model](https://mlsanity.com/paper/2004.09190/) (2020)
- [Multidimensional Scaling, Sammon Mapping, and Isomap: Tutorial and Survey](https://mlsanity.com/paper/2009.08136/) (2020)
- [Unsupervised Depth Estimation, 3D Face Rotation and Replacement](https://mlsanity.com/paper/1803.09202/) (2018)
- [A Simple, Fast and Highly-Accurate Algorithm to Recover 3D Shape from 2D Landmarks on a Single Image](https://mlsanity.com/paper/1609.09058/) (2016)
- [Unified Multi-Modal Landmark Tracking for Tightly Coupled Lidar-Visual-Inertial Odometry](https://mlsanity.com/paper/2011.06838/) (2020)
- [MobileFace: 3D Face Reconstruction with Efficient CNN Regression](https://mlsanity.com/paper/1809.08809/) (2018)
- [On the Exact Recovery Conditions of 3D Human Motion from 2D Landmark Motion with Sparse Articulated Motion](https://mlsanity.com/paper/1907.03967/) (2019)
- [3D Facial Expression Reconstruction using Cascaded Regression](https://mlsanity.com/paper/1712.03491/) (2017)
- [Virtual Landmark-Based Control of Docking Support for Assistive Mobility Devices](https://mlsanity.com/paper/2107.13365/) (2021)

## Abstract (arXiv)

In this paper, a low parameter deep learning framework utilizing the Non-metric Multi-Dimensional scaling (NMDS) method, is proposed to recover the 3D shape of 2D landmarks on a human face, in a single input image. Hence, NMDS approach is used for the first time to establish a mapping from a 2D landmark space to the corresponding 3D shape space. A deep neural network learns the pairwise dissimilarity among 2D landmarks, used by NMDS approach, whose objective is to learn the pairwise 3D Euclidean distance of the corresponding 2D landmarks on the input image. This scheme results in a symmetric dissimilarity matrix, with the rank larger than 2, leading the NMDS approach toward appropriately recovering the 3D shape of corresponding 2D landmarks. In the case of posed images and complex image formation processes like perspective projection which causes occlusion in the input image, we consider an autoencoder component in the proposed framework, as an occlusion removal part, which turns different input views of the human face into a profile view. The results of a performance evaluation using different synthetic and real-world human face datasets, including Besel Face Model (BFM), CelebA, CoMA - FLAME, and CASIA-3D, indicates the comparable performance of the proposed framework, despite its small number of training parameters, with the related state-of-the-art and powerful 3D reconstruction methods from the literature, in terms of efficiency and accuracy.