# Uni-ISP: Toward Unifying the Learning of ISPs from Multiple Mobile Cameras

arXiv: https://arxiv.org/abs/2406.01003 · HTML: https://mlsanity.com/paper/2406.01003/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [HINet: Half Instance Normalization Network for Image Restoration](https://mlsanity.com/paper/2105.06086/) (2021)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [Stripformer: Strip Transformer for Fast Image Deblurring](https://mlsanity.com/paper/2204.04627/) (2022)
- [Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline](https://mlsanity.com/paper/2004.01179/) (2020)
- [Invertible Image Signal Processing](https://mlsanity.com/paper/2103.15061/) (2021)
- [DeepISP: Towards Learning an End-to-End Image Processing Pipeline](https://mlsanity.com/paper/1801.06724/) (2018)
- [Image Aesthetic Assessment: An Experimental Survey](https://mlsanity.com/paper/1610.00838/) (2016)
- [Language-Driven Image Style Transfer](https://mlsanity.com/paper/2106.00178/) (2021)
- [Neural Preset for Color Style Transfer](https://mlsanity.com/paper/2303.13511/) (2023)
- [RawHDR: High Dynamic Range Image Reconstruction from a Single Raw Image](https://mlsanity.com/paper/2309.02020/) (2023)
- [Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer](https://mlsanity.com/paper/2004.10955/) (2020)
- [NILUT: Conditional Neural Implicit 3D Lookup Tables for Image Enhancement](https://mlsanity.com/paper/2306.11920/) (2023)
- [CIE XYZ Net: Unprocessing Images for Low-Level Computer Vision Tasks](https://mlsanity.com/paper/2006.12709/) (2020)
- [ParamISP: Learned Forward and Inverse ISPs using Camera Parameters](https://mlsanity.com/paper/2312.13313/) (2023)
- [Raw Image Deblurring](https://mlsanity.com/paper/2012.04264/) (2020)
- [MetaISP -- Exploiting Global Scene Structure for Accurate Multi-Device Color Rendition](https://mlsanity.com/paper/2401.03220/) (2024)

## Abstract (arXiv)

Modern end-to-end image signal processors (ISPs) can learn complex mappings from RAW/XYZ data to sRGB (and vice versa), opening new possibilities in image processing. However, the growing diversity of camera models, particularly in mobile devices, renders the development of individual ISPs unsustainable due to their limited versatility and adaptability across varied camera systems. In this paper, we introduce Uni-ISP, a novel pipeline that unifies ISP learning for diverse mobile cameras, delivering a highly accurate and adaptable processor. The core of Uni-ISP is leveraging device-aware embeddings through learning forward/inverse ISPs and its special training scheme. By doing so, Uni-ISP not only improves the performance of forward and inverse ISPs but also unlocks new applications previously inaccessible to conventional learned ISPs. To support this work, we construct a real-world 4K dataset, FiveCam, comprising more than 2,400 pairs of sRGB-RAW images captured synchronously by five smartphone cameras. Extensive experiments validate Uni-ISP's accuracy in learning forward and inverse ISPs (with improvements of +2.4dB/1.5dB PSNR), versatility in enabling new applications, and adaptability to new camera models.