# NViSII: A Scriptable Tool for Photorealistic Image Generation

arXiv: https://arxiv.org/abs/2105.13962 · HTML: https://mlsanity.com/paper/2105.13962/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 26 of 26)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation](https://mlsanity.com/paper/1804.06534/) (2018)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task](https://mlsanity.com/paper/1707.02267/) (2017)
- [Rendering of Eyes for Eye-Shape Registration and Gaze Estimation](https://mlsanity.com/paper/1505.05916/) (2015)
- [UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World](https://mlsanity.com/paper/2003.10608/) (2020)
- [UnrealCV: Connecting Computer Vision to Unreal Engine](https://mlsanity.com/paper/1609.01326/) (2016)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection](https://mlsanity.com/paper/1902.09967/) (2019)
- [UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision](https://mlsanity.com/paper/1612.04647/) (2016)
- [Learning to Predict Robot Keypoints Using Artificially Generated Images](https://mlsanity.com/paper/1907.01879/) (2019)

## Built on by (most cited first, 24 of 24)

- [Variable Bitrate Neural Fields](https://mlsanity.com/paper/2206.07707/) (2022)
- [6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark](https://mlsanity.com/paper/2203.05701/) (2022)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021)
- [Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects](https://mlsanity.com/paper/2112.11347/) (2021)
- [StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects](https://mlsanity.com/paper/2211.04604/) (2022)
- [Unity Perception: Generate Synthetic Data for Computer Vision](https://mlsanity.com/paper/2107.04259/) (2021)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis](https://mlsanity.com/paper/2205.07058/) (2022)
- [PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models](https://mlsanity.com/paper/2207.05025/) (2022)
- [PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision](https://mlsanity.com/paper/2112.09290/) (2021)
- [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation](https://mlsanity.com/paper/2407.12449/) (2024)
- [Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement](https://mlsanity.com/paper/2210.00215/) (2022)
- [One-Shot Neural Fields for 3D Object Understanding](https://mlsanity.com/paper/2210.12126/) (2022)
- [WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users](https://mlsanity.com/paper/2404.17063/) (2024)
- [Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environment](https://mlsanity.com/paper/2311.11039/) (2023)
- [Synthetic imagery for fuzzy object detection: A comparative study](https://mlsanity.com/paper/2410.01124/) (2024)
- [SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision](https://mlsanity.com/paper/2505.11439/) (2025)
- [MINERVAS: Massive INterior EnviRonments VirtuAl Synthesis](https://mlsanity.com/paper/2107.06149/) (2021)
- [Using Game Engines and Machine Learning to Create Synthetic Satellite Imagery for a Tabletop Verification Exercise](https://mlsanity.com/paper/2404.11461/) (2024)
- [DEF-oriCORN: efficient 3D scene understanding for robust language-directed manipulation without demonstrations](https://mlsanity.com/paper/2407.21267/) (2024)
- [OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGB](https://mlsanity.com/paper/2410.06694/) (2024)
- [Uncertainty-Resilient Active Intention Recognition for Robotic Assistants](https://mlsanity.com/paper/2508.19150/) (2025)
- [CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image](https://mlsanity.com/paper/2609.25654/) (2026)

## Abstract (arXiv)

We present a Python-based renderer built on NVIDIA's OptiX ray tracing engine and the OptiX AI denoiser, designed to generate high-quality synthetic images for research in computer vision and deep learning. Our tool enables the description and manipulation of complex dynamic 3D scenes containing object meshes, materials, textures, lighting, volumetric data (e.g., smoke), and backgrounds. Metadata, such as 2D/3D bounding boxes, segmentation masks, depth maps, normal maps, material properties, and optical flow vectors, can also be generated. In this work, we discuss design goals, architecture, and performance. We demonstrate the use of data generated by path tracing for training an object detector and pose estimator, showing improved performance in sim-to-real transfer in situations that are difficult for traditional raster-based renderers. We offer this tool as an easy-to-use, performant, high-quality renderer for advancing research in synthetic data generation and deep learning.