# Joint Learning of Intrinsic Images and Semantic Segmentation

arXiv: https://arxiv.org/abs/1807.11857 · HTML: https://mlsanity.com/paper/1807.11857/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2018.

## Built on by (most cited first, 26 of 26)

- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [Zero-Shot Day-Night Domain Adaptation with a Physics Prior](https://mlsanity.com/paper/2108.05137/) (2021)
- [Colorful Diffuse Intrinsic Image Decomposition in the Wild](https://mlsanity.com/paper/2409.13690/) (2024)
- [Generalized Contrastive Optimization of Siamese Networks for Place Recognition](https://mlsanity.com/paper/2103.06638/) (2021)
- [PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition](https://mlsanity.com/paper/2203.16670/) (2022)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [Disentangle then Parse:Night-time Semantic Segmentation with Illumination Disentanglement](https://mlsanity.com/paper/2307.09362/) (2023)
- [DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Rendering](https://mlsanity.com/paper/2507.03924/) (2025)
- [Complementary Feature Enhanced Network with Vision Transformer for Image Dehazing](https://mlsanity.com/paper/2109.07100/) (2021)
- [Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory](https://mlsanity.com/paper/2407.20785/) (2024)
- [Joint Learning of Portrait Intrinsic Decomposition and Relighting](https://mlsanity.com/paper/2106.15305/) (2021)
- [DPF: Learning Dense Prediction Fields with Weak Supervision](https://mlsanity.com/paper/2303.16890/) (2023)
- [SIGNet: Intrinsic Image Decomposition by a Semantic and Invariant Gradient Driven Network for Indoor Scenes](https://mlsanity.com/paper/2208.14369/) (2022)
- [Generalized Task-Driven Medical Image Quality Enhancement with Gradient Promotion](https://mlsanity.com/paper/2501.01114/) (2025)
- [FlowIID: Single-Step Intrinsic Image Decomposition via Latent Flow Matching](https://mlsanity.com/paper/2601.12329/) (2026)
- [Learning Intrinsic Images for Clothing](https://mlsanity.com/paper/2111.08521/) (2021)
- [Invariant Descriptors for Intrinsic Reflectance Optimization](https://mlsanity.com/paper/2204.04076/) (2022)
- [Intrinsic Image Decomposition Using Point Cloud Representation](https://mlsanity.com/paper/2307.10924/) (2023)
- [Neural Implicit Field Editing Considering Object-environment Interaction](https://mlsanity.com/paper/2311.00425/) (2023)
- [A Unified Simulation Framework for Visual and Behavioral Fidelity in Crowd Analysis](https://mlsanity.com/paper/2312.02613/) (2023)
- [Unsupervised Intrinsic Image Decomposition with LiDAR Intensity Enhanced Training](https://mlsanity.com/paper/2403.14089/) (2024)
- [Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics](https://mlsanity.com/paper/2602.01391/) (2026)
- [Olbedo: An Albedo and Shading Aerial Dataset for Large-Scale Outdoor Environments](https://mlsanity.com/paper/2602.22025/) (2026)
- [The frame-level leakage trap: rethinking evaluation protocols for intrinsic image decomposition, with source-separable uncertainty as a case study](https://mlsanity.com/paper/2605.06359/) (2026)

## Abstract (arXiv)

Semantic segmentation of outdoor scenes is problematic when there are variations in imaging conditions. It is known that albedo (reflectance) is invariant to all kinds of illumination effects. Thus, using reflectance images for semantic segmentation task can be favorable. Additionally, not only segmentation may benefit from reflectance, but also segmentation may be useful for reflectance computation. Therefore, in this paper, the tasks of semantic segmentation and intrinsic image decomposition are considered as a combined process by exploring their mutual relationship in a joint fashion. To that end, we propose a supervised end-to-end CNN architecture to jointly learn intrinsic image decomposition and semantic segmentation. We analyze the gains of addressing those two problems jointly. Moreover, new cascade CNN architectures for intrinsic-for-segmentation and segmentation-for-intrinsic are proposed as single tasks. Furthermore, a dataset of 35K synthetic images of natural environments is created with corresponding albedo and shading (intrinsics), as well as semantic labels (segmentation) assigned to each object/scene. The experiments show that joint learning of intrinsic image decomposition and semantic segmentation is beneficial for both tasks for natural scenes. Dataset and models are available at: https://ivi.fnwi.uva.nl/cv/intrinseg