# Cameras as Rays: Pose Estimation via Ray Diffusion

arXiv: https://arxiv.org/abs/2402.14817 · HTML: https://mlsanity.com/paper/2402.14817/

## Key facts

- Cited by 131 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction](https://mlsanity.com/paper/2212.00792/) (2022)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2206.05737/) (2022)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://mlsanity.com/paper/2311.12024/) (2023)
- [PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment](https://mlsanity.com/paper/2306.15667/) (2023)
- [NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild](https://mlsanity.com/paper/2110.07604/) (2021)
- [RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild](https://mlsanity.com/paper/2208.05963/) (2022)
- [RelPose++: Recovering 6D Poses from Sparse-view Observations](https://mlsanity.com/paper/2305.04926/) (2023)
- [SparsePose: Sparse-View Camera Pose Regression and Refinement](https://mlsanity.com/paper/2211.16991/) (2022)
- [Few-View Object Reconstruction with Unknown Categories and Camera Poses](https://mlsanity.com/paper/2212.04492/) (2022)
- [Wide-Baseline Relative Camera Pose Estimation with Directional Learning](https://mlsanity.com/paper/2106.03336/) (2021)
- [RANSAC-Flow: generic two-stage image alignment](https://mlsanity.com/paper/2004.01526/) (2020)

## Built on by (most cited first, 30 of 131)

- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [Global Structure-from-Motion Revisited](https://mlsanity.com/paper/2407.20219/) (2024)
- [FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views](https://mlsanity.com/paper/2502.12138/) (2025)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Image](https://mlsanity.com/paper/2406.04343/) (2024)
- [Cameras as Relative Positional Encoding](https://mlsanity.com/paper/2507.10496/) (2025)
- [SpatialTrackerV2: 3D Point Tracking Made Easy](https://mlsanity.com/paper/2507.12462/) (2025)
- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction](https://mlsanity.com/paper/2504.07961/) (2025)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [Dens3R: A Foundation Model for 3D Geometry Prediction](https://mlsanity.com/paper/2507.16290/) (2025)
- [One Diffusion to Generate Them All](https://mlsanity.com/paper/2411.16318/) (2024)
- [FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction](https://mlsanity.com/paper/2412.09573/) (2024)
- [Matrix3D: Large Photogrammetry Model All-in-One](https://mlsanity.com/paper/2502.07685/) (2025)
- [DeepVerse: 4D Autoregressive Video Generation as a World Model](https://mlsanity.com/paper/2506.01103/) (2025)
- [AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis](https://mlsanity.com/paper/2504.13157/) (2025)
- [SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis](https://mlsanity.com/paper/2411.16443/) (2024)
- [Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation](https://mlsanity.com/paper/2412.21117/) (2024)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [Robust Incremental Structure-from-Motion with Hybrid Features](https://mlsanity.com/paper/2409.19811/) (2024)
- [FlowR: Flowing from Sparse to Dense 3D Reconstructions](https://mlsanity.com/paper/2504.01647/) (2025)
- [Particulate: Feed-Forward 3D Object Articulation](https://mlsanity.com/paper/2512.11798/) (2025)

## Abstract (arXiv)

Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views (<10). In contrast to existing approaches that pursue top-down prediction of global parametrizations of camera extrinsics, we propose a distributed representation of camera pose that treats a camera as a bundle of rays. This representation allows for a tight coupling with spatial image features improving pose precision. We observe that this representation is naturally suited for set-level transformers and develop a regression-based approach that maps image patches to corresponding rays. To capture the inherent uncertainties in sparse-view pose inference, we adapt this approach to learn a denoising diffusion model which allows us to sample plausible modes while improving performance. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D while generalizing to unseen object categories and in-the-wild captures.