# POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

arXiv: https://arxiv.org/abs/2604.06938 · HTML: https://mlsanity.com/paper/2604.06938/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Invertible Image Signal Processing](https://mlsanity.com/paper/2103.15061/) (2021)
- [Boltzmann Exploration Done Right](https://mlsanity.com/paper/1705.10257/) (2017)
- [YOLOv13: Real-Time Object Detection with Hypergraph-Enhanced Adaptive Visual Perception](https://mlsanity.com/paper/2506.17733/) (2025)
- [RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation](https://mlsanity.com/paper/1812.02646/) (2018)
- [Instance Segmentation in the Dark](https://mlsanity.com/paper/2304.14298/) (2023)
- [Model-Based Image Signal Processors via Learnable Dictionaries](https://mlsanity.com/paper/2201.03210/) (2022)
- [ReconfigISP: Reconfigurable Camera Image Processing Pipeline](https://mlsanity.com/paper/2109.04760/) (2021)
- [CIE XYZ Net: Unprocessing Images for Low-Level Computer Vision Tasks](https://mlsanity.com/paper/2006.12709/) (2020)
- [ParamISP: Learned Forward and Inverse ISPs using Camera Parameters](https://mlsanity.com/paper/2312.13313/) (2023)
- [AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection](https://mlsanity.com/paper/2410.22939/) (2024)
- [DRL-ISP: Multi-Objective Camera ISP with Deep Reinforcement Learning](https://mlsanity.com/paper/2207.03081/) (2022)
- [Task-Aware Image Signal Processor for Advanced Visual Perception](https://mlsanity.com/paper/2509.13762/) (2025)

## Built on by (most cited first, 1 of 1)

- [RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulation](https://mlsanity.com/paper/2609.37530/) (2026)

## Abstract (arXiv)

Recent work has explored optimizing image signal processing (ISP) pipelines for various tasks by composing predefined modules and adapting them to task-specific objectives. However, jointly optimizing module sequences and parameters remains challenging. Existing approaches rely on neural architecture search (NAS) or step-wise reinforcement learning (RL), but NAS suffers from a training-inference mismatch, while step-wise RL leads to unstable training and high computational overhead due to stage-wise decision-making. We propose POS-ISP, a sequence-level RL framework that formulates modular ISP optimization as a global sequence prediction problem. Our method predicts the entire module sequence and its parameters in a single forward pass and optimizes the pipeline using a terminal task reward, eliminating the need for intermediate supervision and redundant executions. Experiments across multiple downstream tasks show that POS-ISP improves task performance while reducing computational cost, highlighting sequence-level optimization as a stable and efficient paradigm for task-aware ISP. The project page is available at https://w1jyun.github.io/POS-ISP