# Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence

arXiv: https://arxiv.org/abs/2506.08220 · HTML: https://mlsanity.com/paper/2506.08220/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 66 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 30 of 53)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence](https://mlsanity.com/paper/2305.14334/) (2023)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [SPair-71k: A Large-scale Benchmark for Semantic Correspondence](https://mlsanity.com/paper/1908.10543/) (2019)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence](https://mlsanity.com/paper/2311.17034/) (2023)
- [Continuous Surface Embeddings](https://mlsanity.com/paper/2011.12438/) (2020)
- [CATs: Cost Aggregation Transformers for Visual Correspondence](https://mlsanity.com/paper/2106.02520/) (2021)
- [Canonical Surface Mapping via Geometric Cycle Consistency](https://mlsanity.com/paper/1907.10043/) (2019)
- [Proposal Flow](https://mlsanity.com/paper/1511.05065/) (2015)
- [Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features](https://mlsanity.com/paper/2311.17024/) (2023)
- [Learning to Compose Hypercolumns for Visual Correspondence](https://mlsanity.com/paper/2007.10587/) (2020)
- [CATs++: Boosting Cost Aggregation with Convolutions and Transformers](https://mlsanity.com/paper/2202.06817/) (2022)
- [SCNet: Learning Semantic Correspondence](https://mlsanity.com/paper/1705.04043/) (2017)

## Built on by (most cited first, 6 of 6)

- [Towards Geometry-Grounded Dense Semantic Matching with VGGT Priors](https://mlsanity.com/paper/2509.21263/) (2025)
- [Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence](https://mlsanity.com/paper/2605.30093/) (2026)
- [Quantum-Inspired Harmonic Decision Models: A Computational Framework for Music Generation](https://mlsanity.com/paper/2607.05007/) (2026)
- [SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models](https://mlsanity.com/paper/2605.31597/) (2026)
- [ImCorr: Sub-pixel Semantic Correspondence via Implicit Feature Decoding](https://mlsanity.com/paper/2609.29193/) (2026)
- [Self-Supervised Learning of Semantically Consistent Visual Features via a Simple 3D Prior](https://mlsanity.com/paper/2610.16988/) (2026)

## Abstract (arXiv)

Semantic correspondence (SC) aims to establish semantically meaningful matches across different instances of an object category. We illustrate how recent supervised SC methods remain limited in their ability to generalize beyond sparsely annotated training keypoints, effectively acting as keypoint detectors. To address this, we propose a novel approach for learning dense correspondences by lifting 2D keypoints into a canonical 3D space using monocular depth estimation. Our method constructs a continuous canonical manifold that captures object geometry without requiring explicit 3D supervision or camera annotations. Additionally, we introduce SPair-U, an extension of SPair-71k with novel keypoint annotations, to better assess generalization. Experiments not only demonstrate that our model significantly outperforms supervised baselines on unseen keypoints, highlighting its effectiveness in learning robust correspondences, but that unsupervised baselines outperform supervised counterparts when generalized across different datasets.