# Decomposer: Semi-supervised Learning of Image Restoration and Image Decomposition

arXiv: https://arxiv.org/abs/2311.16829 · HTML: https://mlsanity.com/paper/2311.16829/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 6 of 6)

- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Unsupervised Learning for Intrinsic Image Decomposition from a Single Image](https://mlsanity.com/paper/1911.09930/) (2019)
- [Self-Supervised Intrinsic Image Decomposition](https://mlsanity.com/paper/1711.03678/) (2017)
- [A Deep Convolutional Neural Network for Background Subtraction](https://mlsanity.com/paper/1702.01731/) (2017)
- [Dynamic Background Subtraction by Generative Neural Networks](https://mlsanity.com/paper/2202.05336/) (2022)

## Abstract (arXiv)

We present Decomposer, a semi-supervised reconstruction model that decomposes distorted image sequences into their fundamental building blocks - the original image and the applied augmentations, i.e., shadow, light, and occlusions. To solve this problem, we use the SIDAR dataset that provides a large number of distorted image sequences: each sequence contains images with shadows, lighting, and occlusions applied to an undistorted version. Each distortion changes the original signal in different ways, e.g., additive or multiplicative noise. We propose a transformer-based model to explicitly learn this decomposition. The sequential model uses 3D Swin-Transformers for spatio-temporal encoding and 3D U-Nets as prediction heads for individual parts of the decomposition. We demonstrate that by separately pre-training our model on weakly supervised pseudo labels, we can steer our model to optimize for our ambiguous problem definition and learn to differentiate between the different image distortions.