mlsanityData to 14 Oct 2026

NViSII: A Scriptable Tool for Photorealistic Image Generation

arXiv 2105.13962 ↗First posted 28 May 2021cs.CV cs.ROcited by 24

Not on the map. Why?

Abstract (arXiv)

We present a Python-based renderer built on NVIDIA's OptiX ray tracing engine and the OptiX AI denoiser, designed to generate high-quality synthetic images for research in computer vision and deep learning. Our tool enables the description and manipulation of complex dynamic 3D scenes containing object meshes, materials, textures, lighting, volumetric data (e.g., smoke), and backgrounds. Metadata, such as 2D/3D bounding boxes, segmentation masks, depth maps, normal maps, material properties, and optical flow vectors, can also be generated. In this work, we discuss design goals, architecture, and performance. We demonstrate the use of data generated by path tracing for training an object detector and pose estimator, showing improved performance in sim-to-real transfer in situations that are difficult for traditional raster-based renderers. We offer this tool as an easy-to-use, performant, high-quality renderer for advancing research in synthetic data generation and deep learning.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

26 of 41 references.

References · 26

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  2. 2019Objects as Pointscited by 1,175
  3. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  4. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  6. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  8. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  9. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  10. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  11. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  12. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 294
14 more
  1. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
  2. 2018Unity: A General Platform for Intelligent Agentscited by 236
  3. 2019ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulationcited by 124
  4. 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
  5. 2018Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimationcited by 92
  6. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 90
  7. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  8. 2015Rendering of Eyes for Eye-Shape Registration and Gaze Estimationcited by 49
  9. 2020UnrealText: Synthesizing Realistic Scene Text Images from the Unreal Worldcited by 43
  10. 2016UnrealCV: Connecting Computer Vision to Unreal Enginecited by 41
  11. 2019The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentationcited by 39
  12. 2019An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detectioncited by 11
  13. 2016UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Visioncited by 8
  14. 2019Learning to Predict Robot Keypoints Using Artificially Generated Imagescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 9; 2023: 1; 2024: 6; 2025: 2; 2026: 1.

24 papers.

Sort:

By area:

  1. 2022Variable Bitrate Neural Fieldscited by 117
  2. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 86
  3. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  4. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  5. 2021Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objectscited by 50
  6. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  7. 2021Unity Perception: Generate Synthetic Data for Computer Visioncited by 43
  8. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  9. 2022RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesiscited by 33
  10. 2022PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Modelscited by 15
  11. 2021PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Visioncited by 11
  12. 2024Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulationcited by 6
12 more
  1. 2022Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placementcited by 5
  2. 2022One-Shot Neural Fields for 3D Object Understandingcited by 5
  3. 2024WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Userscited by 5
  4. 2023Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environmentcited by 4
  5. 2024Synthetic imagery for fuzzy object detection: A comparative studycited by 2
  6. 2025SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Visioncited by 2
  7. 2021MINERVAS: Massive INterior EnviRonments VirtuAl Synthesiscited by 1
  8. 2024Using Game Engines and Machine Learning to Create Synthetic Satellite Imagery for a Tabletop Verification Exercisecited by 1
  9. 2024DEF-oriCORN: efficient 3D scene understanding for robust language-directed manipulation without demonstrationscited by 1
  10. 2024OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGBcited by 0
  11. 2025Uncertainty-Resilient Active Intention Recognition for Robotic Assistantscited by 0
  12. 2026CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Imagecited by 0