# Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process

arXiv: https://arxiv.org/abs/2208.07735 · HTML: https://mlsanity.com/paper/2208.07735/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 126 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions](https://mlsanity.com/paper/2111.14813/) (2021)
- [Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining](https://mlsanity.com/paper/1807.05698/) (2018)
- [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017)
- [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning](https://mlsanity.com/paper/1904.05050/) (2019)
- [Patch-Based Image Inpainting with Generative Adversarial Networks](https://mlsanity.com/paper/1803.07422/) (2018)
- [Syn2Real Transfer Learning for Image Deraining using Gaussian Processes](https://mlsanity.com/paper/2006.05580/) (2020)
- [You Only Look Yourself: Unsupervised and Untrained Single Image Dehazing Neural Network](https://mlsanity.com/paper/2006.16829/) (2020)
- [6G Wireless Systems: Vision, Requirements, Challenges, Insights, and Opportunities](https://mlsanity.com/paper/2008.03213/) (2020)
- [Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining](https://mlsanity.com/paper/1808.01491/) (2018)
- [V4D:4D Convolutional Neural Networks for Video-level Representation Learning](https://mlsanity.com/paper/2002.07442/) (2020)
- [Beyond Monocular Deraining: Parallel Stereo Deraining Network Via Semantic Prior](https://mlsanity.com/paper/2105.03830/) (2021)
- [Feature-Aligned Video Raindrop Removal with Temporal Constraints](https://mlsanity.com/paper/2205.14574/) (2022)
- [Massive MIMO Extensions to the COST 2100 Channel Model: Modeling and Validation](https://mlsanity.com/paper/1905.04931/) (2019)
- [Near-field Ultra-wideband mmWave Channel Characterization Using Successive Cancellation Beamspace UCA Algorithm](https://mlsanity.com/paper/1901.10508/) (2019)

## Built on by (most cited first, 4 of 4)

- [RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering](https://mlsanity.com/paper/2404.11401/) (2024)
- [Enhancing Underwater Imaging with 4-D Light Fields: Dataset and Method](https://mlsanity.com/paper/2408.17339/) (2024)
- [MDeRainNet: An Efficient Macro-pixel Image Rain Removal Network](https://mlsanity.com/paper/2406.10652/) (2024)
- [Enhancing Underwater Light Field Images via Global Geometry-aware Diffusion Process](https://mlsanity.com/paper/2601.21179/) (2026)

## Abstract (arXiv)

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.