# LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models

arXiv: https://arxiv.org/abs/2312.01027 · HTML: https://mlsanity.com/paper/2312.01027/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 2 Dec 2023.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Retinex Decomposition for Low-Light Enhancement](https://mlsanity.com/paper/1808.04560/) (2018)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Learning Enriched Features for Real Image Restoration and Enhancement](https://mlsanity.com/paper/2003.06792/) (2020)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Generative Diffusion Prior for Unified Image Restoration and Enhancement](https://mlsanity.com/paper/2304.01247/) (2023)
- [Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation](https://mlsanity.com/paper/2103.00860/) (2021)
- [Low-Light Image and Video Enhancement Using Deep Learning: A Survey](https://mlsanity.com/paper/2104.10729/) (2021)
- [Low-Light Image Enhancement with Wavelet-based Diffusion Models](https://mlsanity.com/paper/2306.00306/) (2023)
- [Diff-Retinex: Rethinking Low-light Image Enhancement with A Generative Diffusion Model](https://mlsanity.com/paper/2308.13164/) (2023)
- [Invertible Image Signal Processing](https://mlsanity.com/paper/2103.15061/) (2021)
- [DeepISP: Towards Learning an End-to-End Image Processing Pipeline](https://mlsanity.com/paper/1801.06724/) (2018)
- [Physics-based Noise Modeling for Extreme Low-light Photography](https://mlsanity.com/paper/2108.02158/) (2021)
- [Learning RAW-to-sRGB Mappings with Inaccurately Aligned Supervision](https://mlsanity.com/paper/2108.08119/) (2021)
- [Abandoning the Bayer-Filter to See in the Dark](https://mlsanity.com/paper/2203.04042/) (2022)
- [CameraNet: A Two-Stage Framework for Effective Camera ISP Learning](https://mlsanity.com/paper/1908.01481/) (2019)
- [ExposureDiffusion: Learning to Expose for Low-light Image Enhancement](https://mlsanity.com/paper/2307.07710/) (2023)
- [Towards Low Light Enhancement with RAW Images](https://mlsanity.com/paper/2112.14022/) (2021)
- [Towards General Low-Light Raw Noise Synthesis and Modeling](https://mlsanity.com/paper/2307.16508/) (2023)

## Built on by (most cited first, 3 of 3)

- [Learning to See in the Extremely Dark](https://mlsanity.com/paper/2506.21132/) (2025)
- [DarkDiff: Advancing Low-Light Raw Enhancement by Retasking Diffusion Models for Camera ISP](https://mlsanity.com/paper/2505.23743/) (2025)
- [HVI-CIDNet+: Beyond Extreme Darkness for Low-Light Image Enhancement](https://mlsanity.com/paper/2507.06814/) (2025)

## Abstract (arXiv)

Enhancing a low-light noisy RAW image into a well-exposed and clean sRGB image is a significant challenge for modern digital cameras. Prior approaches have difficulties in recovering fine-grained details and true colors of the scene under extremely low-light environments due to near-to-zero SNR. Meanwhile, diffusion models have shown significant progress towards general domain image generation. In this paper, we propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules to inject the RAW information into the diffusion denoising process via modulating the intermediate features of UNet. We further observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance. Through extensive experiments on representative datasets, we demonstrate our simple design not only achieves state-of-the-art performance in quantitative evaluations but also shows significant superiority in visual comparisons over strong baselines, which highlight the effectiveness of powerful generative priors for neural ISP under extremely low-light environments. The project page is available at https://csqiangwen.github.io/projects/ldm-isp/