# Event Camera Demosaicing via Swin Transformer and Pixel-focus Loss

arXiv: https://arxiv.org/abs/2404.02731 · HTML: https://mlsanity.com/paper/2404.02731/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021)
- [Replacing Mobile Camera ISP with a Single Deep Learning Model](https://mlsanity.com/paper/2002.05509/) (2020)
- [Deep Image Deblurring: A Survey](https://mlsanity.com/paper/2201.10700/) (2022)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Learning Event-Based Motion Deblurring](https://mlsanity.com/paper/2004.05794/) (2020)
- [Motion Deblurring with Real Events](https://mlsanity.com/paper/2109.13695/) (2021)
- [N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution](https://mlsanity.com/paper/2211.11436/) (2022)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [Multi-Bracket High Dynamic Range Imaging with Event Cameras](https://mlsanity.com/paper/2203.06622/) (2022)
- [Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution](https://mlsanity.com/paper/2303.13767/) (2023)
- [Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks](https://mlsanity.com/paper/1803.05215/) (2018)
- [Beyond Joint Demosaicking and Denoising: An Image Processing Pipeline for a Pixel-bin Image Sensor](https://mlsanity.com/paper/2104.09398/) (2021)

## Built on by (most cited first, 3 of 3)

- [Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing](https://mlsanity.com/paper/2503.16134/) (2025)
- [EmambaIR: Efficient Visual State Space Model for Event-guided Image Reconstruction](https://mlsanity.com/paper/2605.08073/) (2026)
- [Hybrid Event Frame Sensors: Modeling, Calibration, and Simulation](https://mlsanity.com/paper/2511.18037/) (2025)

## Abstract (arXiv)

Recent research has highlighted improvements in high-quality imaging guided by event cameras, with most of these efforts concentrating on the RGB domain. However, these advancements frequently neglect the unique challenges introduced by the inherent flaws in the sensor design of event cameras in the RAW domain. Specifically, this sensor design results in the partial loss of pixel values, posing new challenges for RAW domain processes like demosaicing. The challenge intensifies as most research in the RAW domain is based on the premise that each pixel contains a value, making the straightforward adaptation of these methods to event camera demosaicing problematic. To end this, we present a Swin-Transformer-based backbone and a pixel-focus loss function for demosaicing with missing pixel values in RAW domain processing. Our core motivation is to refine a general and widely applicable foundational model from the RGB domain for RAW domain processing, thereby broadening the model's applicability within the entire imaging process. Our method harnesses multi-scale processing and space-to-depth techniques to ensure efficiency and reduce computing complexity. We also proposed the Pixel-focus Loss function for network fine-tuning to improve network convergence based on our discovery of a long-tailed distribution in training loss. Our method has undergone validation on the MIPI Demosaic Challenge dataset, with subsequent analytical experimentation confirming its efficacy. All code and trained models are released here: https://github.com/yunfanLu/ev-demosaic