# Deep Direct Volume Rendering: Learning Visual Feature Mappings From Exemplary Images

arXiv: https://arxiv.org/abs/2106.05429 · HTML: https://mlsanity.com/paper/2106.05429/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Real-Time User-Guided Image Colorization with Learned Deep Priors](https://mlsanity.com/paper/1705.02999/) (2017)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [RenderNet: A deep convolutional network for differentiable rendering from 3D shapes](https://mlsanity.com/paper/1806.06575/) (2018)
- [InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations](https://mlsanity.com/paper/1908.00407/) (2019)
- [A Generative Model for Volume Rendering](https://mlsanity.com/paper/1710.09545/) (2017)
- [Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool](https://mlsanity.com/paper/1912.04591/) (2019)
- [Volumetric Isosurface Rendering with Deep Learning-Based Super-Resolution](https://mlsanity.com/paper/1906.06520/) (2019)
- [Deep Volumetric Ambient Occlusion](https://mlsanity.com/paper/2008.08345/) (2020)

## Built on by (most cited first, 4 of 4)

- [DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualization](https://mlsanity.com/paper/2204.06504/) (2022)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [ParamsDrag: Interactive Parameter Space Exploration via Image-Space Dragging](https://mlsanity.com/paper/2407.14100/) (2024)
- [Monocular Depth Decomposition of Semi-Transparent Volume Renderings](https://mlsanity.com/paper/2206.13282/) (2022)

## Abstract (arXiv)

Volume Rendering is an important technique for visualizing three-dimensional scalar data grids and is commonly employed for scientific and medical image data. Direct Volume Rendering (DVR) is a well established and efficient rendering algorithm for volumetric data. Neural rendering uses deep neural networks to solve inverse rendering tasks and applies techniques similar to DVR. However, it has not been demonstrated successfully for the rendering of scientific volume data. In this work, we introduce Deep Direct Volume Rendering (DeepDVR), a generalization of DVR that allows for the integration of deep neural networks into the DVR algorithm. We conceptualize the rendering in a latent color space, thus enabling the use of deep architectures to learn implicit mappings for feature extraction and classification, replacing explicit feature design and hand-crafted transfer functions. Our generalization serves to derive novel volume rendering architectures that can be trained end-to-end directly from examples in image space, obviating the need to manually define and fine-tune multidimensional transfer functions while providing superior classification strength. We further introduce a novel stepsize annealing scheme to accelerate the training of DeepDVR models and validate its effectiveness in a set of experiments. We validate our architectures on two example use cases: (1) learning an optimized rendering from manually adjusted reference images for a single volume and (2) learning advanced visualization concepts like shading and semantic colorization that generalize to unseen volume data. We find that deep volume rendering architectures with explicit modeling of the DVR pipeline effectively enable end-to-end learning of scientific volume rendering tasks from target images.