# Dirty Pixels: Towards End-to-End Image Processing and Perception

arXiv: https://arxiv.org/abs/1701.06487 · HTML: https://mlsanity.com/paper/1701.06487/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2017.

## Built on by (most cited first, 30 of 42)

- [Instance Segmentation in the Dark](https://mlsanity.com/paper/2304.14298/) (2023)
- [RobustSAM: Segment Anything Robustly on Degraded Images](https://mlsanity.com/paper/2406.09627/) (2024)
- [DynamicISP: Dynamically Controlled Image Signal Processor for Image Recognition](https://mlsanity.com/paper/2211.01146/) (2022)
- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images](https://mlsanity.com/paper/2408.14802/) (2024)
- [Rawgment: Noise-Accounted RAW Augmentation Enables Recognition in a Wide Variety of Environments](https://mlsanity.com/paper/2210.16046/) (2022)
- [Efficient Visual Computing with Camera RAW Snapshots](https://mlsanity.com/paper/2212.07778/) (2022)
- [Towards RAW Object Detection in Diverse Conditions](https://mlsanity.com/paper/2411.15678/) (2024)
- [ISP Distillation](https://mlsanity.com/paper/2101.10203/) (2021)
- [Computational Imaging for Machine Perception: Transferring Semantic Segmentation beyond Aberrations](https://mlsanity.com/paper/2211.11257/) (2022)
- [A Vision-Centric Approach for Static Map Element Annotation](https://mlsanity.com/paper/2309.11754/) (2023)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors](https://mlsanity.com/paper/2307.10667/) (2023)
- [All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines](https://mlsanity.com/paper/2112.09219/) (2021)
- [Dark-ISP: Enhancing RAW Image Processing for Low-Light Object Detection](https://mlsanity.com/paper/2509.09183/) (2025)
- [Enhancing Video Analytics Accuracy via Real-time Automated Camera Parameter Tuning](https://mlsanity.com/paper/2107.03964/) (2021)
- [Dynamic Test-Time Augmentation via Differentiable Functions](https://mlsanity.com/paper/2212.04681/) (2022)
- [CAMAv2: A Vision-Centric Approach for Static Map Element Annotation](https://mlsanity.com/paper/2407.21331/) (2024)
- [Robust 3D Object Detection using Probabilistic Point Clouds from Single-Photon LiDARs](https://mlsanity.com/paper/2508.00169/) (2025)
- [RAWild: Sensor-Agnostic RAW Object Detection via Physics-Guided Curve and Grid Modeling](https://mlsanity.com/paper/2605.05941/) (2026)
- [Test-time Adaptation Meets Image Enhancement: Improving Accuracy via Uncertainty-aware Logit Switching](https://mlsanity.com/paper/2403.17423/) (2024)
- [TaCOS: Task-Specific Camera Optimization with Simulation](https://mlsanity.com/paper/2404.11031/) (2024)
- [SimROD: A Simple Baseline for Raw Object Detection with Global and Local Enhancements](https://mlsanity.com/paper/2503.07101/) (2025)
- [RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images and A Benchmark](https://mlsanity.com/paper/2503.17027/) (2025)
- [Task-Aware Image Signal Processor for Advanced Visual Perception](https://mlsanity.com/paper/2509.13762/) (2025)
- [γ-Quant: Towards Learnable Quantization for Low-bit Pattern Recognition](https://mlsanity.com/paper/2509.22448/) (2025)
- [Differentiable Microscopy for Content and Task Aware Compressive Fluorescence Imaging](https://mlsanity.com/paper/2203.14945/) (2022)
- [GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation](https://mlsanity.com/paper/2506.02882/) (2025)
- [RAWDet-7: A Multi-Scenario Benchmark for Object Detection and Description on Quantized RAW Images](https://mlsanity.com/paper/2602.03760/) (2026)
- [UniISP: A Unified ISP Framework for Both Human and Machine Vision](https://mlsanity.com/paper/2605.07359/) (2026)

## Abstract (arXiv)

Real-world imaging systems acquire measurements that are degraded by noise, optical aberrations, and other imperfections that make image processing for human viewing and higher-level perception tasks challenging. Conventional cameras address this problem by compartmentalizing imaging from high-level task processing. As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping and compression. This pipeline is optimized to obtain a visually pleasing image. High-level processing, on the other hand, involves steps such as feature extraction, classification, tracking, and fusion. While this siloed design approach allows for efficient development, it also dictates compartmentalized performance metrics, without knowledge of the higher-level task of the camera system. For example, today's demosaicking and denoising algorithms are designed using perceptual image quality metrics but not with domain-specific tasks such as object detection in mind. We propose an end-to-end differentiable architecture that jointly performs demosaicking, denoising, deblurring, tone-mapping, and classification. The architecture learns processing pipelines whose outputs differ from those of existing ISPs optimized for perceptual quality, preserving fine detail at the cost of increased noise and artifacts. We demonstrate on captured and simulated data that our model substantially improves perception in low light and other challenging conditions, which is imperative for real-world applications. Finally, we found that the proposed model also achieves state-of-the-art accuracy when optimized for image reconstruction in low-light conditions, validating the architecture itself as a potentially useful drop-in network for reconstruction and analysis tasks beyond the applications demonstrated in this work.