# IGNOR: Image-guided Neural Object Rendering

arXiv: https://arxiv.org/abs/1811.10720 · HTML: https://mlsanity.com/paper/1811.10720/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Light Field Neural Rendering](https://mlsanity.com/paper/2112.09687/) (2021)
- [MonoHuman: Animatable Human Neural Field from Monocular Video](https://mlsanity.com/paper/2304.02001/) (2023)
- [Neural Mesh-Based Graphics](https://mlsanity.com/paper/2208.05785/) (2022)
- [Free-Viewpoint RGB-D Human Performance Capture and Rendering](https://mlsanity.com/paper/2112.13889/) (2021)
- [HDhuman: High-quality Human Novel-view Rendering from Sparse Views](https://mlsanity.com/paper/2201.08158/) (2022)
- [Progressively-connected Light Field Network for Efficient View Synthesis](https://mlsanity.com/paper/2207.04465/) (2022)
- [PackUV: Packed Gaussian UV Maps for 4D Volumetric Video](https://mlsanity.com/paper/2602.23040/) (2026)
- [Rethinking Directional Integration in Neural Radiance Fields](https://mlsanity.com/paper/2311.16504/) (2023)
- [Capabilities, Limitations and Challenges of Style Transfer with CycleGANs: A Study on Automatic Ring Design Generation](https://mlsanity.com/paper/2207.08989/) (2022)
- [GMT: Enhancing Generalizable Neural Rendering via Geometry-Driven Multi-Reference Texture Transfer](https://mlsanity.com/paper/2410.00672/) (2024)

## Abstract (arXiv)

We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.