# Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference

arXiv: https://arxiv.org/abs/2008.10710 · HTML: https://mlsanity.com/paper/2008.10710/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [Perception-Oriented Video Frame Interpolation via Asymmetric Blending](https://mlsanity.com/paper/2404.06692/) (2024)
- [Real-RawVSR: Real-World Raw Video Super-Resolution with a Benchmark Dataset](https://mlsanity.com/paper/2209.12475/) (2022)
- [AccFlow: Backward Accumulation for Long-Range Optical Flow](https://mlsanity.com/paper/2308.13133/) (2023)
- [Learning for Unconstrained Space-Time Video Super-Resolution](https://mlsanity.com/paper/2102.13011/) (2021)
- [FastLLVE: Real-Time Low-Light Video Enhancement with Intensity-Aware Lookup Table](https://mlsanity.com/paper/2308.06749/) (2023)
- [GridDehazeNet+: An Enhanced Multi-Scale Network with Intra-Task Knowledge Transfer for Single Image Dehazing](https://mlsanity.com/paper/2103.13998/) (2021)
- [Light-VQA: A Multi-Dimensional Quality Assessment Model for Low-Light Video Enhancement](https://mlsanity.com/paper/2305.09512/) (2023)
- [Recaptured Raw Screen Image and Video Demoir'eing via Channel and Spatial Modulations](https://mlsanity.com/paper/2310.20332/) (2023)
- [TransMRSR: Transformer-based Self-Distilled Generative Prior for Brain MRI Super-Resolution](https://mlsanity.com/paper/2306.06669/) (2023)
- [HDR Video Reconstruction with a Large Dynamic Dataset in Raw and sRGB Domains](https://mlsanity.com/paper/2304.04773/) (2023)
- [NTIRE 2023 Quality Assessment of Video Enhancement Challenge](https://mlsanity.com/paper/2307.09729/) (2023)
- [Light-VQA+: A Video Quality Assessment Model for Exposure Correction with Vision-Language Guidance](https://mlsanity.com/paper/2405.03333/) (2024)
- [Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation](https://mlsanity.com/paper/2403.06452/) (2024)
- [A2-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Masks](https://mlsanity.com/paper/2603.10685/) (2026)
- [FUMO: Prior-Modulated Diffusion for Single Image Reflection Removal](https://mlsanity.com/paper/2603.19036/) (2026)
- [Resolution-Agnostic Neural Compression for High-Fidelity Portrait Video Conferencing via Implicit Radiance Fields](https://mlsanity.com/paper/2402.16599/) (2024)
- [Compression-Realized Deep Structural Network for Video Quality Enhancement](https://mlsanity.com/paper/2405.06342/) (2024)

## Abstract (arXiv)

To the best of our knowledge, the existing deep-learning-based Video Super-Resolution (VSR) methods exclusively make use of videos produced by the Image Signal Processor (ISP) of the camera system as inputs. Such methods are 1) inherently suboptimal due to information loss incurred by non-invertible operations in ISP, and 2) inconsistent with the real imaging pipeline where VSR in fact serves as a pre-processing unit of ISP. To address this issue, we propose a new VSR method that can directly exploit camera sensor data, accompanied by a carefully built Raw Video Dataset (RawVD) for training, validation, and testing. This method consists of a Successive Deep Inference (SDI) module and a reconstruction module, among others. The SDI module is designed according to the architectural principle suggested by a canonical decomposition result for Hidden Markov Model (HMM) inference; it estimates the target high-resolution frame by repeatedly performing pairwise feature fusion using deformable convolutions. The reconstruction module, built with elaborately designed Attention-based Residual Dense Blocks (ARDBs), serves the purpose of 1) refining the fused feature and 2) learning the color information needed to generate a spatial-specific transformation for accurate color correction. Extensive experiments demonstrate that owing to the informativeness of the camera raw data, the effectiveness of the network architecture, and the separation of super-resolution and color correction processes, the proposed method achieves superior VSR results compared to the state-of-the-art and can be adapted to any specific camera-ISP. Code and dataset are available at https://github.com/proteus1991/RawVSR.