mlsanityData to 7 Oct 2026

3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue

arXiv 2211.14823 ↗First posted 27 Nov 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. Due to the technical difficulty, one can only obtain rough 3D models (R3DMs) for most real objects using existing 3D reconstruction techniques. As a result, physically-based rendering (PBR) would render low-quality images or videos for scenes that are constructed by R3DMs. One promising solution would be representing real-world objects as Neural Fields such as NeRFs, which are able to generate photo-realistic renderings of an object under desired viewpoints. However, a drawback is that the synthesized views through Neural Fields Rendering (NFR) cannot reflect the simulated lighting details on R3DMs in PBR pipelines, especially when object interactions in the 3D scene creation cause local shadows. To solve this dilemma, we propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other. LighTNet reasons about a simplified image composition model, remedies the uneven surface issue caused by R3DMs, and is empowered by several perceptual-motivated constraints and a new Lab angle loss which enhances the contrast between lighting strength and colors. Comparisons demonstrate that LighTNet is superior in synthesizing impressive lighting, and is promising in pushing NFR further in practical 3D modeling workflows.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 56 of its 82 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

56 of 82 references.

References · 56

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  8. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  10. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  11. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  12. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
38 more
  1. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  2. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  4. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  5. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  6. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  7. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  8. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  9. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  10. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  11. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  12. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  13. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  14. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  15. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  16. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  17. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  18. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  19. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  20. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  21. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  22. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
  23. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  24. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  25. 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
  26. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  27. 2022Modeling Indirect Illumination for Inverse Renderingcited by 167
  28. 2019Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Imagescited by 164
  29. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 161
  30. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  31. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121
  32. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  33. 2022EfficientNeRF: Efficient Neural Radiance Fieldscited by 110
  34. 2021Light Field Neural Renderingcited by 108
  35. 2019Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networkscited by 106
  36. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  37. 2019Learning Physics-guided Face Relighting under Directional Lightcited by 90
  38. 2019Controlling Neural Level Setscited by 76

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.