# RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

arXiv: https://arxiv.org/abs/2205.07058 · HTML: https://mlsanity.com/paper/2205.07058/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 93 references that are in the map.
- First posted on arXiv 14 May 2022.

## Builds on (most cited first, 30 of 65)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)

## Built on by (most cited first, 30 of 33)

- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS](https://mlsanity.com/paper/2312.04564/) (2023)
- [Variable Bitrate Neural Fields](https://mlsanity.com/paper/2206.07707/) (2022)
- [EscherNet: A Generative Model for Scalable View Synthesis](https://mlsanity.com/paper/2402.03908/) (2024)
- [Consistent123: Improve Consistency for One Image to 3D Object Synthesis](https://mlsanity.com/paper/2310.08092/) (2023)
- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes](https://mlsanity.com/paper/2308.12967/) (2023)
- [Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review](https://mlsanity.com/paper/2405.03417/) (2024)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Views](https://mlsanity.com/paper/2312.04424/) (2023)
- [SHACIRA: Scalable HAsh-grid Compression for Implicit Neural Representations](https://mlsanity.com/paper/2309.15848/) (2023)
- [TOSS:High-quality Text-guided Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2310.10644/) (2023)
- [iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis](https://mlsanity.com/paper/2310.16167/) (2023)
- [SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images](https://mlsanity.com/paper/2212.09100/) (2022)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion](https://mlsanity.com/paper/2501.18804/) (2025)
- [3DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models](https://mlsanity.com/paper/2503.21745/) (2025)
- [Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering](https://mlsanity.com/paper/2405.18677/) (2024)
- [ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimization](https://mlsanity.com/paper/2311.14208/) (2023)
- [SEED4D: A Synthetic Ego--Exo Dynamic 4D Data Generator, Driving Dataset and Benchmark](https://mlsanity.com/paper/2412.00730/) (2024)
- [Scaling Sequence-to-Sequence Generative Neural Rendering](https://mlsanity.com/paper/2510.04236/) (2025)
- [Content-Aware Radiance Fields: Aligning Model Complexity with Scene Intricacy Through Learned Bitwidth Quantization](https://mlsanity.com/paper/2410.19483/) (2024)
- [VaLID: Variable-Length Input Diffusion for Novel View Synthesis](https://mlsanity.com/paper/2312.08892/) (2023)
- [Distilling Multi-view Diffusion Models into 3D Generators](https://mlsanity.com/paper/2504.00457/) (2025)
- [Light Field Diffusion for Single-View Novel View Synthesis](https://mlsanity.com/paper/2309.11525/) (2023)
- [Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcription](https://mlsanity.com/paper/2403.10953/) (2024)
- [NVGS: Neural Visibility for Occlusion Culling in 3D Gaussian Splatting](https://mlsanity.com/paper/2511.19202/) (2025)
- [Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizers](https://mlsanity.com/paper/2503.13588/) (2025)

## Abstract (arXiv)

We present a large-scale synthetic dataset for novel view synthesis consisting of ~300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution (1600 x 1600 pixels). The dataset is orders of magnitude larger than existing synthetic datasets for novel view synthesis, thus providing a large unified benchmark for both training and evaluation. Using 4 distinct sources of high-quality 3D meshes, the scenes of our dataset exhibit challenging variations in camera views, lighting, shape, materials, and textures. Because our dataset is too large for existing methods to process, we propose Sparse Voxel Light Field (SVLF), an efficient voxel-based light field approach for novel view synthesis that achieves comparable performance to NeRF on synthetic data, while being an order of magnitude faster to train and two orders of magnitude faster to render. SVLF achieves this speed by relying on a sparse voxel octree, careful voxel sampling (requiring only a handful of queries per ray), and reduced network structure; as well as ground truth depth maps at training time. Our dataset is generated by NViSII, a Python-based ray tracing renderer, which is designed to be simple for non-experts to use and share, flexible and powerful through its use of scripting, and able to create high-quality and physically-based rendered images. Experiments with a subset of our dataset allow us to compare standard methods like NeRF and mip-NeRF for single-scene modeling, and pixelNeRF for category-level modeling, pointing toward the need for future improvements in this area.