# Neural Semantic Surface Maps

arXiv: https://arxiv.org/abs/2309.04836 · HTML: https://mlsanity.com/paper/2309.04836/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 94 references that are in the map.
- First posted on arXiv 9 Sep 2023.

## Builds on (most cited first, 30 of 40)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [PPFNet: Global Context Aware Local Features for Robust 3D Point Matching](https://mlsanity.com/paper/1802.02669/) (2018)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Deep Functional Maps: Structured Prediction for Dense Shape Correspondence](https://mlsanity.com/paper/1704.08686/) (2017)
- [Deep Geometric Functional Maps: Robust Feature Learning for Shape Correspondence](https://mlsanity.com/paper/2003.14286/) (2020)
- [Virtual Multi-view Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2007.13138/) (2020)
- [Continuous and Orientation-preserving Correspondences via Functional Maps](https://mlsanity.com/paper/1806.04455/) (2018)
- [Smooth Shells: Multi-Scale Shape Registration with Functional Maps](https://mlsanity.com/paper/1905.12512/) (2019)
- [Self-supervised Learning of Point Clouds via Orientation Estimation](https://mlsanity.com/paper/2008.00305/) (2020)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [3D Shape Segmentation with Projective Convolutional Networks](https://mlsanity.com/paper/1612.02808/) (2016)
- [SATR: Zero-Shot Semantic Segmentation of 3D Shapes](https://mlsanity.com/paper/2304.04909/) (2023)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Building Statistical Shape Spaces for 3D Human Modeling](https://mlsanity.com/paper/1503.05860/) (2015)

## Built on by (most cited first, 13 of 13)

- [Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features](https://mlsanity.com/paper/2311.18113/) (2023)
- [SHIC: Shape-Image Correspondences with no Keypoint Supervision](https://mlsanity.com/paper/2407.18907/) (2024)
- [NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs](https://mlsanity.com/paper/2402.08622/) (2024)
- [Textured 3D Regenerative Morphing with 3D Diffusion Prior](https://mlsanity.com/paper/2502.14316/) (2025)
- [Dynamic Neural Surfaces for Elastic 4D Shape Representation and Analysis](https://mlsanity.com/paper/2503.03132/) (2025)
- [Surface-Aware Distilled 3D Semantic Features](https://mlsanity.com/paper/2503.18254/) (2025)
- [RARE: Refine Any Registration of Pairwise Point Clouds via Zero-Shot Learning](https://mlsanity.com/paper/2507.19950/) (2025)
- [Learning 3D Scene Analogies with Neural Contextual Scene Maps](https://mlsanity.com/paper/2503.15897/) (2025)
- [MorphAny3D: Unleashing the Power of Structured Latent in 3D Morphing](https://mlsanity.com/paper/2601.00204/) (2026)
- [Non-Rigid 3D Shape Correspondences: From Foundations to Open Challenges and Opportunities](https://mlsanity.com/paper/2604.01274/) (2026)
- [Wukong's 72 Transformations: High-fidelity Textured 3D Morphing via Flow Models](https://mlsanity.com/paper/2511.22425/) (2025)
- [SGMatch: Semantic-Guided Non-Rigid Shape Matching with Flow Regularization](https://mlsanity.com/paper/2603.12937/) (2026)
- [Recent Advances and Trends in Learning-based 3D Representations](https://mlsanity.com/paper/2606.04871/) (2026)

## Abstract (arXiv)

We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Lack of annotated data prohibits direct inference of 3D semantic priors; instead, current State-of-the-art methods predominantly optimize geometric properties or require varying amounts of manual annotation. To overcome the lack of annotated training data, we distill semantic matches from pre-trained vision models: our method renders the pair of 3D shapes from multiple viewpoints; the resulting renders are then fed into an off-the-shelf image-matching method which leverages a pretrained visual model to produce feature points. This yields semantic correspondences, which can be projected back to the 3D shapes, producing a raw matching that is inaccurate and inconsistent between different viewpoints. These correspondences are refined and distilled into an inter-surface map by a dedicated optimization scheme, which promotes bijectivity and continuity of the output map. We illustrate that our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement. Furthermore, it proves effective in scenarios with high semantic complexity, where objects are non-isometrically related, as well as in situations where they are nearly isometric.