mlsanityData to 14 Oct 2026

SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections

arXiv 2205.15768 ↗First posted 31 May 2022cs.CV cs.GR cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

Inverse rendering of an object under entirely unknown capture conditions is a fundamental challenge in computer vision and graphics. Neural approaches such as NeRF have achieved photorealistic results on novel view synthesis, but they require known camera poses. Solving this problem with unknown camera poses is highly challenging as it requires joint optimization over shape, radiance, and pose. This problem is exacerbated when the input images are captured in the wild with varying backgrounds and illuminations. Standard pose estimation techniques fail in such image collections in the wild due to very few estimated correspondences across images. Furthermore, NeRF cannot relight a scene under any illumination, as it operates on radiance (the product of reflectance and illumination). We propose a joint optimization framework to estimate the shape, BRDF, and per-image camera pose and illumination. Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR. To our knowledge, our method is the first to tackle this severely unconstrained task with minimal user interaction. Project page: https://markboss.me/publication/2022-samurai/ Video: https://youtu.be/LlYuGDjXp-8

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 46 of its 81 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

46 of 81 references.

References · 46

  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  4. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  5. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  6. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  8. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  9. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  10. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  11. 2020Neural Sparse Voxel Fieldscited by 959
  12. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
34 more
  1. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  2. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  3. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 646
  4. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  5. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  6. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  7. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 468
  8. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 449
  9. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 412
  10. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
  11. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 335
  12. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 205
  13. 2021Self-Calibrating Neural Radiance Fieldscited by 204
  14. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 203
  15. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  16. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  17. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  18. 2021NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wildcited by 136
  19. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  20. 2021Neural Lumigraph Renderingcited by 105
  21. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  22. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  23. 2020Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Imagescited by 98
  24. 2020Shape and Viewpoint without Keypointscited by 92
  25. 2018Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Imagecited by 66
  26. 2020Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Imagescited by 65
  27. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  28. 2018Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networkscited by 51
  29. 2019Flexible SVBRDF Capture with a Multi-Image Deep Networkcited by 50
  30. 2021Generative Modelling of BRDF Textures from Flash Imagescited by 35
  31. 2020Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfacescited by 33
  32. 2020Guided Fine-Tuning for Large-Scale Material Transfercited by 27
  33. 2020Deep SVBRDF Estimation on Real Materialscited by 15
  34. 2019Single Image BRDF Parameter Estimation with a Conditional Adversarial Networkcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 33; 2024: 27; 2025: 13; 2026: 9.

89 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  3. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 221
  4. 2022SPARF: Neural Radiance Fields from Sparse and Noisy Posescited by 195
  5. 2023NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Imagescited by 163
  6. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 144
  7. 2023Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracingcited by 142
  8. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  9. 2023RobustNeRF: Ignoring Distractors with Robust Lossescited by 101
  10. 2024Neural Gaffer: Relighting Any Object via Diffusioncited by 91
  11. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 90
  12. 2024Fully Explicit Dynamic Gaussian Splattingcited by 76
38 more
  1. 2022LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discoverycited by 66
  2. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  3. 2022SparsePose: Sparse-View Camera Pose Regression and Refinementcited by 53
  4. 2023NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshescited by 53
  5. 2023NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and Pose Annotationscited by 53
  6. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  7. 2023Relighting Neural Radiance Fields with Shadow and Highlight Hintscited by 42
  8. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  9. 2023LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFscited by 33
  10. 2022Hi-LASSIE: High-Fidelity Articulated Shape and Skeleton Discovery from Sparse Image Ensemblecited by 31
  11. 2025Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Datasetcited by 29
  12. 2023ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Renderingcited by 28
  13. 2023Relightable and Animatable Neural Avatar from Sparse-View Videocited by 28
  14. 2024Bilateral Guided Radiance Field Processingcited by 28
  15. 2022In-Hand 3D Object Scanning from an RGB Sequencecited by 27
  16. 2023ASIC: Aligning Sparse in-the-wild Image Collectionscited by 27
  17. 2023CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulationcited by 26
  18. 2023NeISF: Neural Incident Stokes Field for Geometry and Material Estimationcited by 25
  19. 2024RelitLRM: Generative Relightable Radiance for Large Reconstruction Modelscited by 23
  20. 2023ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collectionscited by 22
  21. 2023Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracingcited by 22
  22. 2023FaceLit: Neural 3D Relightable Facescited by 20
  23. 2023SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenescited by 19
  24. 2024SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wildcited by 18
  25. 2024Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splattingcited by 18
  26. 2025LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fieldscited by 16
  27. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  28. 2023UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fieldscited by 12
  29. 2024Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Videocited by 12
  30. 2023CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learningcited by 11
  31. 2024Blind Inversion using Latent Diffusion Priorscited by 10
  32. 2024Edify 3D: Scalable High-Quality 3D Asset Generationcited by 10
  33. 2023Geometry-biased Transformers for Novel View Synthesiscited by 9
  34. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 9
  35. 2023SC-NeuS: Consistent Neural Surface Reconstruction from Sparse and Noisy Viewscited by 8
  36. 2024NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Renderingcited by 8
  37. 2024UniCal: Unified Neural Sensor Calibrationcited by 8
  38. 2023TraM-NeRF: Tracing Mirror and Near-Perfect Specular Reflections through Neural Radiance Fieldscited by 7

50 most cited of 89 All 89 (JSON)