# Generic 3D Representation via Pose Estimation and Matching

arXiv: https://arxiv.org/abs/1710.08247 · HTML: https://mlsanity.com/paper/1710.08247/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2017.

## Built on by (most cited first, 17 of 17)

- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation](https://mlsanity.com/paper/2108.11550/) (2021)
- [Joint-Task Regularization for Partially Labeled Multi-Task Learning](https://mlsanity.com/paper/2404.01976/) (2024)
- [MuM: Multi-View Masked Image Modeling for 3D Vision](https://mlsanity.com/paper/2511.17309/) (2025)
- [Feature Pyramid Network for Multi-task Affective Analysis](https://mlsanity.com/paper/2107.03670/) (2021)
- [Self-Ordering Point Clouds](https://mlsanity.com/paper/2304.00961/) (2023)
- [RCNet: Deep Recurrent Collaborative Network for Multi-View Low-Light Image Enhancement](https://mlsanity.com/paper/2409.04363/) (2024)
- [Simple Control Baselines for Evaluating Transfer Learning](https://mlsanity.com/paper/2202.03365/) (2022)
- [JEDI: Joint Expert Distillation in a Semi-Supervised Multi-Dataset Student-Teacher Scenario for Video Action Recognition](https://mlsanity.com/paper/2308.04934/) (2023)
- [MIMIC: Masked Image Modeling with Image Correspondences](https://mlsanity.com/paper/2306.15128/) (2023)
- [DMTG: One-Shot Differentiable Multi-Task Grouping](https://mlsanity.com/paper/2407.05082/) (2024)
- [HeatMat: Simulation of City Material Impact on Urban Heat Island Effect](https://mlsanity.com/paper/2601.22796/) (2026)
- [LoD-Loc v3: Generalized Aerial Localization in Dense Cities using Instance Silhouette Alignment](https://mlsanity.com/paper/2603.19609/) (2026)
- [Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos](https://mlsanity.com/paper/2606.07366/) (2026)
- [FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation](https://mlsanity.com/paper/2609.11486/) (2026)

## Abstract (arXiv)

Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited. In this paper, we learn a generic 3D representation through solving a set of foundational proxy 3D tasks: object-centric camera pose estimation and wide baseline feature matching. Our method is based upon the premise that by providing supervision over a set of carefully selected foundational tasks, generalization to novel tasks and abstraction capabilities can be achieved. We empirically show that the internal representation of a multi-task ConvNet trained to solve the above core problems generalizes to novel 3D tasks (e.g., scene layout estimation, object pose estimation, surface normal estimation) without the need for fine-tuning and shows traits of abstraction abilities (e.g., cross-modality pose estimation). In the context of the core supervised tasks, we demonstrate our representation achieves state-of-the-art wide baseline feature matching results without requiring apriori rectification (unlike SIFT and the majority of learned features). We also show 6DOF camera pose estimation given a pair local image patches. The accuracy of both supervised tasks come comparable to humans. Finally, we contribute a large-scale dataset composed of object-centric street view scenes along with point correspondences and camera pose information, and conclude with a discussion on the learned representation and open research questions.