# RayZer: A Self-supervised Large View Synthesis Model

arXiv: https://arxiv.org/abs/2505.00702 · HTML: https://mlsanity.com/paper/2505.00702/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 101 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 30 of 84)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)

## Built on by (most cited first, 30 of 77)

- [Unified Camera Positional Encoding for Controlled Video Generation](https://mlsanity.com/paper/2512.07237/) (2025)
- [UFM: A Simple Path towards Unified Dense Correspondence with Flow](https://mlsanity.com/paper/2506.09278/) (2025)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training](https://mlsanity.com/paper/2603.04385/) (2026)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training](https://mlsanity.com/paper/2512.10950/) (2025)
- [iLRM: An Iterative Large 3D Reconstruction Model](https://mlsanity.com/paper/2507.23277/) (2025)
- [True Self-Supervised Novel View Synthesis is Transferable](https://mlsanity.com/paper/2510.13063/) (2025)
- [Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis](https://mlsanity.com/paper/2511.20186/) (2025)
- [4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation](https://mlsanity.com/paper/2506.18839/) (2025)
- [4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time](https://mlsanity.com/paper/2506.18890/) (2025)
- [Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2512.15508/) (2025)
- [SceneTok: A Compressed, Diffusable Token Space for 3D Scenes](https://mlsanity.com/paper/2602.18882/) (2026)
- [ARTDECO: Towards Efficient and High-Fidelity On-the-Fly 3D Reconstruction with Structured Scene Representation](https://mlsanity.com/paper/2510.08551/) (2025)
- [Scaling View Synthesis Transformers](https://mlsanity.com/paper/2602.21341/) (2026)
- [LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis](https://mlsanity.com/paper/2603.20176/) (2026)
- [Multi-view Pyramid Transformer: Look Coarser to See Broader](https://mlsanity.com/paper/2512.07806/) (2025)
- [NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction](https://mlsanity.com/paper/2603.04179/) (2026)
- [Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning](https://mlsanity.com/paper/2602.21186/) (2026)
- [RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion](https://mlsanity.com/paper/2506.05285/) (2025)
- [From Rays to Projections: Better Inputs for Feed-Forward View Synthesis](https://mlsanity.com/paper/2601.05116/) (2026)
- [SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images](https://mlsanity.com/paper/2510.15072/) (2025)
- [Selfi: Self Improving Reconstruction Engine via 3D Geometric Feature Alignment](https://mlsanity.com/paper/2512.08930/) (2025)
- [WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments](https://mlsanity.com/paper/2601.10716/) (2026)
- [RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space](https://mlsanity.com/paper/2602.20685/) (2026)
- [Reliev3R: Relieving Feed-forward Reconstruction from Multi-View Geometric Annotations](https://mlsanity.com/paper/2604.00548/) (2026)
- [FrameCrafter: Novel View Synthesis as Video Completion](https://mlsanity.com/paper/2604.08500/) (2026)
- [Self-Improving 4D Perception via Self-Distillation](https://mlsanity.com/paper/2604.08532/) (2026)
- [Fast Spatial Memory with Elastic Test-Time Training](https://mlsanity.com/paper/2604.07350/) (2026)

## Abstract (arXiv)

We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than ``oracle'' methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/