# Weak Multi-View Supervision for Surface Mapping Estimation

arXiv: https://arxiv.org/abs/2105.01388 · HTML: https://mlsanity.com/paper/2105.01388/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [Articulation-aware Canonical Surface Mapping](https://mlsanity.com/paper/2004.00614/) (2020)
- [Canonical Surface Mapping via Geometric Cycle Consistency](https://mlsanity.com/paper/1907.10043/) (2019)
- [Category-Specific Object Reconstruction from a Single Image](https://mlsanity.com/paper/1411.6069/) (2014)
- [Multiview Aggregation for Learning Category-Specific Shape Reconstruction](https://mlsanity.com/paper/1907.01085/) (2019)

## Abstract (arXiv)

We propose a weakly-supervised multi-view learning approach to learn category-specific surface mapping without dense annotations. We learn the underlying surface geometry of common categories, such as human faces, cars, and airplanes, given instances from those categories. While traditional approaches solve this problem using extensive supervision in the form of pixel-level annotations, we take advantage of the fact that pixel-level UV and mesh predictions can be combined with 3D reprojections to form consistency cycles. As a result of exploiting these cycles, we can establish a dense correspondence mapping between image pixels and the mesh acting as a self-supervisory signal, which in turn helps improve our overall estimates. Our approach leverages information from multiple views of the object to establish additional consistency cycles, thus improving surface mapping understanding without the need for explicit annotations. We also propose the use of deformation fields for predictions of an instance specific mesh. Given the lack of datasets providing multiple images of similar object instances from different viewpoints, we generate and release a multi-view ShapeNet Cars and Airplanes dataset created by rendering ShapeNet meshes using a 360 degree camera trajectory around the mesh. For the human faces category, we process and adapt an existing dataset to a multi-view setup. Through experimental evaluations, we show that, at test time, our method can generate accurate variations away from the mean shape, is multi-view consistent, and performs comparably to fully supervised approaches.