# Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments

arXiv: https://arxiv.org/abs/2409.11854 · HTML: https://mlsanity.com/paper/2409.11854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration](https://mlsanity.com/paper/1604.01093/) (2016)
- [Neural Inverse Rendering of an Indoor Scene from a Single Image](https://mlsanity.com/paper/1901.02453/) (2019)
- [DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality](https://mlsanity.com/paper/1904.01175/) (2019)
- [Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination](https://mlsanity.com/paper/2003.08367/) (2020)
- [IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2206.08423/) (2022)
- [Neural-PBIR Reconstruction of Shape, Material, and Illumination](https://mlsanity.com/paper/2304.13445/) (2023)
- [Photometric Bundle Adjustment for Vision-Based SLAM](https://mlsanity.com/paper/1608.02026/) (2016)
- [Direct Sparse Mapping](https://mlsanity.com/paper/1904.06577/) (2019)
- [Efficient 3D Deep LiDAR Odometry](https://mlsanity.com/paper/2111.02135/) (2021)
- [Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes](https://mlsanity.com/paper/2211.10206/) (2022)
- [Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry](https://mlsanity.com/paper/2003.00766/) (2020)
- [Sparse Needlets for Lighting Estimation with Spherical Transport Loss](https://mlsanity.com/paper/2106.13090/) (2021)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)

## Built on by (most cited first, 1 of 1)

- [Robust 3DGS-based SLAM via Adaptive Kernel Smoothing](https://mlsanity.com/paper/2511.23221/) (2025)

## Abstract (arXiv)

Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency significantly affects the reliability of existing PBA methods. To solve this problem, we propose a novel physically-based PBA method. Specifically, we introduce the physically-based weights regarding material, illumination, and light path. These weights distinguish the pixel pairs with different levels of photometric inconsistency. We also design corresponding models for material estimation based on sequential images and illumination estimation based on point clouds. In addition, we establish the first SLAM-related dataset of non-Lambertian scenes with complete ground truth of illumination and material. Extensive experiments demonstrated that our PBA method outperforms existing approaches in accuracy.