# Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal Processing

arXiv: https://arxiv.org/abs/2104.00253 · HTML: https://mlsanity.com/paper/2104.00253/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 59 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 34)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Noise2Void - Learning Denoising from Single Noisy Images](https://mlsanity.com/paper/1811.10980/) (2018)
- [DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better](https://mlsanity.com/paper/1908.03826/) (2019)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering](https://mlsanity.com/paper/1610.04794/) (2016)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Benchmarking Denoising Algorithms with Real Photographs](https://mlsanity.com/paper/1707.01313/) (2017)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Noisier2Noise: Learning to Denoise from Unpaired Noisy Data](https://mlsanity.com/paper/1910.11908/) (2019)
- [Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring](https://mlsanity.com/paper/2004.05343/) (2020)
- [Replacing Mobile Camera ISP with a Single Deep Learning Model](https://mlsanity.com/paper/2002.05509/) (2020)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [DeepISP: Towards Learning an End-to-End Image Processing Pipeline](https://mlsanity.com/paper/1801.06724/) (2018)
- [NBNet: Noise Basis Learning for Image Denoising with Subspace Projection](https://mlsanity.com/paper/2012.15028/) (2020)
- [Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising](https://mlsanity.com/paper/2010.11971/) (2020)
- [Deep White-Balance Editing](https://mlsanity.com/paper/2004.01354/) (2020)
- [Learning Latent Subspaces in Variational Autoencoders](https://mlsanity.com/paper/1812.06190/) (2018)
- [Denoising Criterion for Variational Auto-Encoding Framework](https://mlsanity.com/paper/1511.06406/) (2015)
- [Deep Generative Adversarial Compression Artifact Removal](https://mlsanity.com/paper/1704.02518/) (2017)
- [Deep clustering: On the link between discriminative models and K-means](https://mlsanity.com/paper/1810.04246/) (2018)
- [Deep Graph-Convolutional Image Denoising](https://mlsanity.com/paper/1907.08448/) (2019)
- [Nonparametric Variational Auto-encoders for Hierarchical Representation Learning](https://mlsanity.com/paper/1703.07027/) (2017)
- [Learning Deformable Kernels for Image and Video Denoising](https://mlsanity.com/paper/1904.06903/) (2019)
- [NLH: A Blind Pixel-level Non-local Method for Real-world Image Denoising](https://mlsanity.com/paper/1906.06834/) (2019)

## Abstract (arXiv)

Camera Image Signal Processing (ISP) pipelines can get appealing results in different image signal processing tasks. Nonetheless, the majority of these methods, including those employing an encoder-decoder deep architecture for the task, typically utilize a uniform filter applied consistently across the entire image. However, it is natural to view a camera image as heterogeneous, as the color intensity and the artificial noise are distributed vastly differently, even across the two-dimensional domain of a single image. Varied Moire ringing, motion blur, color-bleaching, or lens-based projection distortions can all potentially lead to a heterogeneous image artifact filtering problem. In this paper, we present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts (especially image denoising). We call our three-fold deep-trained model the Patch Subspace Learning Autoencoder (PSL-AE). The PSL-AE model does not make assumptions regarding uniform levels of image distortion. Instead, it first encodes patches extracted from noisy a nd clean image pairs, with different artifact types or distortion levels, by contrastive learning. Then, the patches of each image are encoded into corresponding soft clusters within their suitable latent sub-space, utilizing a prior mixture model. Furthermore, the decoders undergo training in an unsupervised manner, specifically trained for the image patches present in each cluster. The experiments highlight the adaptability and efficacy through enhanced heterogeneous filtering, both from synthesized artifacts but also realistic SIDD image pairs.