RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models

The three areas of realistic forward rendering, per-pixel inverse rendering, and generative image synthesis may seem like separate and unrelated sub-fields of graphics and vision. However, recent work has demonstrated improved estimation of per-pixel intrinsic channels (albedo, roughness, metallicity) based on a diffusion architecture; we call this the RGBrightarrowX problem. We further show that the reverse problem of synthesizing realistic images given intrinsic channels, XrightarrowRGB, can also be addressed in a diffusion framework. Focusing on the image domain of interior scenes, we introduce an improved diffusion model for RGBrightarrowX, which also estimates lighting, as well as the first diffusion XrightarrowRGB model capable of synthesizing realistic images from (full or partial) intrinsic channels. Our XrightarrowRGB model explores a middle ground between traditional rendering and generative models: we can specify only certain appearance properties that should be followed, and give freedom to the model to hallucinate a plausible version of the rest. This flexibility makes it possible to use a mix of heterogeneous training datasets, which differ in the available channels. We use multiple existing datasets and extend them with our own synthetic and real data, resulting in a model capable of extracting scene properties better than previous work and of generating highly realistic images of interior scenes.