# Learn Your Scales: Towards Scale-Consistent Generative Novel View Synthesis

arXiv: https://arxiv.org/abs/2503.15412 · HTML: https://mlsanity.com/paper/2503.15412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [Generative Novel View Synthesis with 3D-Aware Diffusion Models](https://mlsanity.com/paper/2304.02602/) (2023)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image](https://mlsanity.com/paper/2012.09855/) (2020)
- [Towards Zero-Shot Scale-Aware Monocular Depth Estimation](https://mlsanity.com/paper/2306.17253/) (2023)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2304.10700/) (2023)
- [Controlling Space and Time with Diffusion Models](https://mlsanity.com/paper/2407.07860/) (2024)
- [Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image](https://mlsanity.com/paper/2203.09457/) (2022)
- [Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model](https://mlsanity.com/paper/2312.13252/) (2023)
- [Learning Facial Representations from the Cycle-consistency of Face](https://mlsanity.com/paper/2108.03427/) (2021)
- [PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis](https://mlsanity.com/paper/2402.17986/) (2024)
- [Accurate and Robust Scale Recovery for Monocular Visual Odometry Based on Plane Geometry](https://mlsanity.com/paper/2101.05995/) (2021)

## Abstract (arXiv)

Conventional depth-free multi-view datasets are captured using a moving monocular camera without metric calibration. The scales of camera positions in this monocular setting are ambiguous. Previous methods have acknowledged scale ambiguity in multi-view data via various ad-hoc normalization pre-processing steps, but have not directly analyzed the effect of incorrect scene scales on their application. In this paper, we seek to understand and address the effect of scale ambiguity when used to train generative novel view synthesis methods (GNVS). In GNVS, new views of a scene or object can be minimally synthesized given a single image and are, thus, unconstrained, necessitating the use of generative methods. The generative nature of these models captures all aspects of uncertainty, including any uncertainty of scene scales, which act as nuisance variables for the task. We study the effect of scene scale ambiguity in GNVS when sampled from a single image by isolating its effect on the resulting models and, based on these intuitions, define new metrics that measure the scale inconsistency of generated views. We then propose a framework to estimate scene scales jointly with the GNVS model in an end-to-end fashion. Empirically, we show that our method reduces the scale inconsistency of generated views without the complexity or downsides of previous scale normalization methods. Further, we show that removing this ambiguity improves generated image quality of the resulting GNVS model.