# Virtual Correspondence: Humans as a Cue for Extreme-View Geometry

arXiv: https://arxiv.org/abs/2206.08365 · HTML: https://mlsanity.com/paper/2206.08365/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 97 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 46)

- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses](https://mlsanity.com/paper/1905.04132/) (2019)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [Learning Feature Descriptors using Camera Pose Supervision](https://mlsanity.com/paper/2004.13324/) (2020)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Universal Correspondence Network](https://mlsanity.com/paper/1606.03558/) (2016)
- [S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling](https://mlsanity.com/paper/2101.06571/) (2021)
- [Human Mesh Recovery from Multiple Shots](https://mlsanity.com/paper/2012.09843/) (2020)
- [Wide-Baseline Relative Camera Pose Estimation with Directional Learning](https://mlsanity.com/paper/2106.03336/) (2021)
- [SCNet: Learning Semantic Correspondence](https://mlsanity.com/paper/1705.04043/) (2017)
- [Proposal Flow: Semantic Correspondences from Object Proposals](https://mlsanity.com/paper/1703.07144/) (2017)
- [Reconstructing 3D Human Pose by Watching Humans in the Mirror](https://mlsanity.com/paper/2104.00340/) (2021)
- [FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence](https://mlsanity.com/paper/1702.00926/) (2017)
- [Extreme Rotation Estimation using Dense Correlation Volumes](https://mlsanity.com/paper/2104.13530/) (2021)
- [Torchreid: A Library for Deep Learning Person Re-Identification in Pytorch](https://mlsanity.com/paper/1910.10093/) (2019)

## Built on by (most cited first, 25 of 25)

- [Visual Geometry Grounded Deep Structure From Motion](https://mlsanity.com/paper/2312.04563/) (2023)
- [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://mlsanity.com/paper/2311.12024/) (2023)
- [PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment](https://mlsanity.com/paper/2306.15667/) (2023)
- [RelPose++: Recovering 6D Poses from Sparse-view Observations](https://mlsanity.com/paper/2305.04926/) (2023)
- [SpatialTrackerV2: 3D Point Tracking Made Easy](https://mlsanity.com/paper/2507.12462/) (2025)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [360-Degree Panorama Generation from Few Unregistered NFoV Images](https://mlsanity.com/paper/2308.14686/) (2023)
- [CASA: Category-agnostic Skeletal Animal Reconstruction](https://mlsanity.com/paper/2211.03568/) (2022)
- [Reconstructing People, Places, and Cameras](https://mlsanity.com/paper/2412.17806/) (2024)
- [From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos](https://mlsanity.com/paper/2412.07770/) (2024)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction](https://mlsanity.com/paper/2309.10748/) (2023)
- [ObjectMatch: Robust Registration using Canonical Object Correspondences](https://mlsanity.com/paper/2212.01985/) (2022)
- [3D Reconstruction of Objects in Hands without Real World 3D Supervision](https://mlsanity.com/paper/2305.03036/) (2023)
- [RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting](https://mlsanity.com/paper/2404.19706/) (2024)
- [LFM-3D: Learnable Feature Matching Across Wide Baselines Using 3D Signals](https://mlsanity.com/paper/2303.12779/) (2023)
- [You Only Look at One: Category-Level Object Representations for Pose Estimation From a Single Example](https://mlsanity.com/paper/2305.12626/) (2023)
- [Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos](https://mlsanity.com/paper/2412.19089/) (2024)
- [HM3D-ABO: A Photo-realistic Dataset for Object-centric Multi-view 3D Reconstruction](https://mlsanity.com/paper/2206.12356/) (2022)
- [Learning-based Relational Object Matching Across Views](https://mlsanity.com/paper/2305.02398/) (2023)
- [Extreme Two-View Geometry From Object Poses with Diffusion Models](https://mlsanity.com/paper/2402.02800/) (2024)
- [Multi-View Person Matching and 3D Pose Estimation with Arbitrary Uncalibrated Camera Networks](https://mlsanity.com/paper/2312.01561/) (2023)
- [Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching](https://mlsanity.com/paper/2411.15860/) (2024)
- [YOWO: You Only Walk Once to Jointly Map An Indoor Scene and Register Ceiling-mounted Cameras](https://mlsanity.com/paper/2511.16521/) (2025)
- [RPGD: RANSAC-P3P Gradient Descent for Extrinsic Calibration in 3D Human Pose Estimation](https://mlsanity.com/paper/2602.13901/) (2026)

## Abstract (arXiv)

Recovering the spatial layout of the cameras and the geometry of the scene from extreme-view images is a longstanding challenge in computer vision. Prevailing 3D reconstruction algorithms often adopt the image matching paradigm and presume that a portion of the scene is co-visible across images, yielding poor performance when there is little overlap among inputs. In contrast, humans can associate visible parts in one image to the corresponding invisible components in another image via prior knowledge of the shapes. Inspired by this fact, we present a novel concept called virtual correspondences (VCs). VCs are a pair of pixels from two images whose camera rays intersect in 3D. Similar to classic correspondences, VCs conform with epipolar geometry; unlike classic correspondences, VCs do not need to be co-visible across views. Therefore VCs can be established and exploited even if images do not overlap. We introduce a method to find virtual correspondences based on humans in the scene. We showcase how VCs can be seamlessly integrated with classic bundle adjustment to recover camera poses across extreme views. Experiments show that our method significantly outperforms state-of-the-art camera pose estimation methods in challenging scenarios and is comparable in the traditional densely captured setup. Our approach also unleashes the potential of multiple downstream tasks such as scene reconstruction from multi-view stereo and novel view synthesis in extreme-view scenarios.