# True Self-Supervised Novel View Synthesis is Transferable

arXiv: https://arxiv.org/abs/2510.13063 · HTML: https://mlsanity.com/paper/2510.13063/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 30 of 31)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [ReCamMaster: Camera-Controlled Generative Rendering from A Single Video](https://mlsanity.com/paper/2503.11647/) (2025)
- [No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images](https://mlsanity.com/paper/2410.24207/) (2024)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [Visual Geometry Grounded Deep Structure From Motion](https://mlsanity.com/paper/2312.04563/) (2023)
- [COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2312.07504/) (2023)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [History-Guided Video Diffusion](https://mlsanity.com/paper/2502.06764/) (2025)
- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields](https://mlsanity.com/paper/2303.14478/) (2023)
- [RUST: Latent Neural Scene Representations from Unposed Imagery](https://mlsanity.com/paper/2211.14306/) (2022)
- [SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splatting](https://mlsanity.com/paper/2411.17190/) (2024)
- [No Pose at All: Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views](https://mlsanity.com/paper/2508.01171/) (2025)
- [FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow](https://mlsanity.com/paper/2306.00180/) (2023)
- [What Do Latent Action Models Actually Learn?](https://mlsanity.com/paper/2506.15691/) (2025)
- [MELON: NeRF with Unposed Images in SO(3)](https://mlsanity.com/paper/2303.08096/) (2023)

## Built on by (most cited first, 15 of 15)

- [E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training](https://mlsanity.com/paper/2512.10950/) (2025)
- [SceneTok: A Compressed, Diffusable Token Space for 3D Scenes](https://mlsanity.com/paper/2602.18882/) (2026)
- [Scaling View Synthesis Transformers](https://mlsanity.com/paper/2602.21341/) (2026)
- [WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments](https://mlsanity.com/paper/2601.10716/) (2026)
- [Fast Spatial Memory with Elastic Test-Time Training](https://mlsanity.com/paper/2604.07350/) (2026)
- [From None to All: Self-Supervised 3D Reconstruction via Novel View Synthesis](https://mlsanity.com/paper/2603.27455/) (2026)
- [Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting](https://mlsanity.com/paper/2604.21776/) (2026)
- [DVSM: Decoder-only View Synthesis Model Done Right](https://mlsanity.com/paper/2605.29891/) (2026)
- [SeeSE3: Emergence of 3D Space in Vision Features](https://mlsanity.com/paper/2607.14228/) (2026)
- [LA-Pose: Latent Action Pretraining Meets Pose Estimation](https://mlsanity.com/paper/2604.27448/) (2026)
- [RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video](https://mlsanity.com/paper/2605.31535/) (2026)
- [Diffuse AI Control on Fuzzy Tasks](https://mlsanity.com/paper/2606.08892/) (2026)
- [Unifying Video Tasks via Spatiotemporal Analogy](https://mlsanity.com/paper/2609.33935/) (2026)
- [Emergent Multi-View Geometry Through Self-Distillation](https://mlsanity.com/paper/2609.39227/) (2026)
- [Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis](https://mlsanity.com/paper/2610.03717/) (2026)

## Abstract (arXiv)

In this paper, we identify that the key criterion for determining whether a model is truly capable of novel view synthesis (NVS) is transferability: Whether any pose representation extracted from one video sequence can be used to re-render the same camera trajectory in another. We analyze prior work on self-supervised NVS and find that their predicted poses do not transfer: The same set of poses lead to different camera trajectories in different 3D scenes. Here, we present XFactor, the first geometry-free self-supervised model capable of true NVS. XFactor combines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. Remarkably, we show that XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry -- such as an explicit parameterization of poses as elements of SE(3). We introduce a new metric to quantify transferability, and through large-scale experiments, we demonstrate that XFactor significantly outperforms prior pose-free NVS transformers, and show that latent poses are highly correlated with real-world poses through probing experiments.