# MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

arXiv: https://arxiv.org/abs/2208.08580 · HTML: https://mlsanity.com/paper/2208.08580/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 49 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Builds on (most cited first, 30 of 39)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [SO-Net: Self-Organizing Network for Point Cloud Analysis](https://mlsanity.com/paper/1803.04249/) (2018)
- [Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts](https://mlsanity.com/paper/2012.09165/) (2020)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline](https://mlsanity.com/paper/2106.05304/) (2021)
- [MVTN: Multi-View Transformation Network for 3D Shape Recognition](https://mlsanity.com/paper/2011.13244/) (2020)
- [3D Point Capsule Networks](https://mlsanity.com/paper/1812.10775/) (2018)
- [Virtual Multi-view Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2007.13138/) (2020)
- [Multiresolution Tree Networks for 3D Point Cloud Processing](https://mlsanity.com/paper/1807.03520/) (2018)
- [BAE-NET: Branched Autoencoder for Shape Co-Segmentation](https://mlsanity.com/paper/1903.11228/) (2019)
- [Unsupervised Multi-Task Feature Learning on Point Clouds](https://mlsanity.com/paper/1910.08207/) (2019)
- [Universal Correspondence Network](https://mlsanity.com/paper/1606.03558/) (2016)
- [3D Shape Segmentation with Projective Convolutional Networks](https://mlsanity.com/paper/1612.02808/) (2016)
- [Pri3D: Can 3D Priors Help 2D Representation Learning?](https://mlsanity.com/paper/2104.11225/) (2021)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)

## Built on by (most cited first, 10 of 10)

- [Unsupervised Semantic Correspondence Using Stable Diffusion](https://mlsanity.com/paper/2305.15581/) (2023)
- [PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models](https://mlsanity.com/paper/2212.01558/) (2022)
- [Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features](https://mlsanity.com/paper/2311.17024/) (2023)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation](https://mlsanity.com/paper/2311.14262/) (2023)
- [CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs](https://mlsanity.com/paper/2311.16703/) (2023)
- [Neural Semantic Surface Maps](https://mlsanity.com/paper/2309.04836/) (2023)
- [3x2: 3D Object Part Segmentation by 2D Semantic Correspondences](https://mlsanity.com/paper/2407.09648/) (2024)
- [PointVST: Self-Supervised Pre-training for 3D Point Clouds via View-Specific Point-to-Image Translation](https://mlsanity.com/paper/2212.14197/) (2022)
- [HAL3D: Hierarchical Active Learning for Fine-Grained 3D Part Labeling](https://mlsanity.com/paper/2301.10460/) (2023)

## Abstract (arXiv)

We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. This is inspired by the observation that view-based surface representations are more effective at modeling high-resolution surface details and texture than their 3D counterparts based on point clouds or voxel occupancy. Specifically, given a 3D shape, we render it from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent, leading to better generalization when trained on a limited number of labeled shapes compared to alternatives that utilize self-supervision in 2D or 3D alone. Experiments on textured (RenderPeople) and untextured (PartNet) 3D datasets show that our method outperforms state-of-the-art alternatives in fine-grained part segmentation. The improvements over baselines are greater when only a sparse set of views is available for training or when shapes are textured, indicating that MvDeCor benefits from both 2D processing and 3D geometric reasoning.