# Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence

arXiv: https://arxiv.org/abs/2311.17034 · HTML: https://mlsanity.com/paper/2311.17034/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence](https://mlsanity.com/paper/2305.14334/) (2023)
- [DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models](https://mlsanity.com/paper/2307.02421/) (2023)
- [SPair-71k: A Large-scale Benchmark for Semantic Correspondence](https://mlsanity.com/paper/1908.10543/) (2019)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [Unsupervised Semantic Correspondence Using Stable Diffusion](https://mlsanity.com/paper/2305.15581/) (2023)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)
- [Learning Accurate Dense Correspondences and When to Trust Them](https://mlsanity.com/paper/2101.01710/) (2021)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [CATs: Cost Aggregation Transformers for Visual Correspondence](https://mlsanity.com/paper/2106.02520/) (2021)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [End-to-end weakly-supervised semantic alignment](https://mlsanity.com/paper/1712.06861/) (2017)
- [SFNet: Learning Object-aware Semantic Correspondence](https://mlsanity.com/paper/1904.01810/) (2019)
- [Proposal Flow](https://mlsanity.com/paper/1511.05065/) (2015)
- [Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence](https://mlsanity.com/paper/2005.05207/) (2020)

## Built on by (most cited first, 30 of 86)

- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)
- [RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation](https://mlsanity.com/paper/2407.04689/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [CleanDIFT: Diffusion Features without Noise](https://mlsanity.com/paper/2412.03439/) (2024)
- [AnyTop: Character Animation Diffusion with Any Topology](https://mlsanity.com/paper/2502.17327/) (2025)
- [Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps](https://mlsanity.com/paper/2312.13216/) (2023)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [Law of Vision Representation in MLLMs](https://mlsanity.com/paper/2408.16357/) (2024)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [EffoVPR: Effective Foundation Model Utilization for Visual Place Recognition](https://mlsanity.com/paper/2405.18065/) (2024)
- [Can Visual Foundation Models Achieve Long-term Point Tracking?](https://mlsanity.com/paper/2408.13575/) (2024)
- [MOSAIC: Multi-Subject Personalized Generation via Correspondence-Aware Alignment and Disentanglement](https://mlsanity.com/paper/2509.01977/) (2025)
- [Toward a Holistic Evaluation of Robustness in CLIP Models](https://mlsanity.com/paper/2410.01534/) (2024)
- [Do It Yourself: Learning Semantic Correspondence from Pseudo-Labels](https://mlsanity.com/paper/2506.05312/) (2025)
- [SHIC: Shape-Image Correspondences with no Keypoint Supervision](https://mlsanity.com/paper/2407.18907/) (2024)
- [MATCHA:Towards Matching Anything](https://mlsanity.com/paper/2501.14945/) (2025)
- [INSID3: Training-Free In-Context Segmentation with DINOv3](https://mlsanity.com/paper/2603.28480/) (2026)
- [Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptors](https://mlsanity.com/paper/2403.14526/) (2024)
- [SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Image](https://mlsanity.com/paper/2409.16178/) (2024)
- [JOG3R: Towards 3D-Consistent Video Generators](https://mlsanity.com/paper/2501.01409/) (2025)
- [RecDreamer: Consistent Text-to-3D Generation via Uniform Score Distillation](https://mlsanity.com/paper/2502.12640/) (2025)
- [Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space](https://mlsanity.com/paper/2504.21749/) (2025)
- [Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models](https://mlsanity.com/paper/2406.07008/) (2024)
- [CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models](https://mlsanity.com/paper/2501.02355/) (2025)
- [SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation](https://mlsanity.com/paper/2505.21795/) (2025)
- [Stable-SCore: A Stable Registration-based Framework for 3D Shape Correspondence](https://mlsanity.com/paper/2503.21766/) (2025)
- [SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations](https://mlsanity.com/paper/2503.22462/) (2025)
- [Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion](https://mlsanity.com/paper/2408.00458/) (2024)
- [Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence](https://mlsanity.com/paper/2506.08220/) (2025)

## Abstract (arXiv)

While pre-trained large-scale vision models have shown significant promise for semantic correspondence, their features often struggle to grasp the geometry and orientation of instances. This paper identifies the importance of being geometry-aware for semantic correspondence and reveals a limitation of the features of current foundation models under simple post-processing. We show that incorporating this information can markedly enhance semantic correspondence performance with simple but effective solutions in both zero-shot and supervised settings. We also construct a new challenging benchmark for semantic correspondence built from an existing animal pose estimation dataset, for both pre-training validating models. Our method achieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on the challenging SPair-71k dataset, outperforming the state of the art by 5.5p and 11.0p absolute gains, respectively. Our code and datasets are publicly available at: https://telling-left-from-right.github.io/.