mlsanityData to 7 Oct 2026

SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild

arXiv 2401.10171 ↗First posted 18 Jan 2024cs.CV cs.GRcited by 18

Not on the map. Why?

Abstract (arXiv)

We present SHINOBI, an end-to-end framework for the reconstruction of shape, material, and illumination from object images captured with varying lighting, pose, and background. Inverse rendering of an object based on unconstrained image collections is a long-standing challenge in computer vision and graphics and requires a joint optimization over shape, radiance, and pose. We show that an implicit shape representation based on a multi-resolution hash encoding enables faster and robust shape reconstruction with joint camera alignment optimization that outperforms prior work. Further, to enable the editing of illumination and object reflectance (i.e. material) we jointly optimize BRDF and illumination together with the object's shape. Our method is class-agnostic and works on in-the-wild image collections of objects to produce relightable 3D assets for several use cases such as AR/VR, movies, games, etc. Project page: https://shinobi.aengelhardt.com Video: https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 68 of its 92 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

68 of 92 references.

References · 68

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  8. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  9. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  11. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  12. 2021Volume Rendering of Neural Implicit Surfacescited by 990
38 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  2. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  3. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  4. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  5. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  6. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  7. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 556
  8. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  9. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  10. 2023Neuralangelo: High-Fidelity Neural Surface Reconstructioncited by 464
  11. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  12. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  13. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
  14. 2022NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstructioncited by 311
  15. 2023FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularizationcited by 310
  16. 2022NoPe-NeRF: Optimising Neural Radiance Field with No Pose Priorcited by 235
  17. 2021Self-Calibrating Neural Radiance Fieldscited by 204
  18. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  19. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  20. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  21. 2022SPARF: Neural Radiance Fields from Sparse and Noisy Posescited by 195
  22. 2022SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Viewscited by 175
  23. 2022Modeling Indirect Illumination for Inverse Renderingcited by 167
  24. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  25. 2023PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustmentcited by 146
  26. 2021NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wildcited by 136
  27. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  28. 2022NeILF: Neural Incident Light Field for Physically-based Material Estimationcited by 105
  29. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  30. 2020Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Imagescited by 98
  31. 2020Shape and Viewpoint without Keypointscited by 92
  32. 2022SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collectionscited by 89
  33. 2021LOLNeRF: Learn from One Lookcited by 84
  34. 2023NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimationcited by 73
  35. 2018Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Imagecited by 66
  36. 2020Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Imagescited by 65
  37. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 63
  38. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collectionscited by 117
  2. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  3. 2025LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fieldscited by 16
  4. 2024Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variationcited by 13
  5. 2024ARM: Appearance Reconstruction Model for Relightable 3D Generationcited by 7
  6. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
  7. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 7
  8. 2024Differentiable Inverse Rendering with Interpretable Basis BRDFscited by 6
  9. 2025Multi-view Surface Reconstruction Using Normal and Reflectance Cuescited by 5
  10. 2024Unveiling the Ambiguity in Neural Inverse Rendering: A Parameter Compensation Analysiscited by 3
  11. 2024Monocular Facial Appearance Capture in the Wildcited by 2
  12. 2025UniVerse: Unleashing the Scene Prior of Video Diffusion Models for Robust Radiance Field Reconstructioncited by 2
6 more
  1. 2026LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windowscited by 2
  2. 2025NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Priorcited by 1
  3. 2026ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illuminationcited by 1
  4. 2024InstantGeoAvatar: Effective Geometry and Appearance Modeling of Animatable Avatars from Monocular Videocited by 0
  5. 2025GLOW: Global Illumination-Aware Inverse Rendering of Indoor Scenes Captured with Dynamic Co-Located Light & Cameracited by 0
  6. 2026Dynamic Inverse Rendering for Enhanced Material-Lighting Decompositioncited by 0