# RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering

arXiv: https://arxiv.org/abs/2404.11401 · HTML: https://mlsanity.com/paper/2404.11401/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining](https://mlsanity.com/paper/1807.05698/) (2018)
- [Learning A Sparse Transformer Network for Effective Image Deraining](https://mlsanity.com/paper/2303.11950/) (2023)
- [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [Deblur-NeRF: Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2111.14292/) (2021)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [A Model-driven Deep Neural Network for Single Image Rain Removal](https://mlsanity.com/paper/2005.01333/) (2020)
- [HDR-NeRF: High Dynamic Range Neural Radiance Fields](https://mlsanity.com/paper/2111.14451/) (2021)
- [Syn2Real Transfer Learning for Image Deraining using Gaussian Processes](https://mlsanity.com/paper/2006.05580/) (2020)
- [Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework](https://mlsanity.com/paper/1803.10433/) (2018)
- [NAN: Noise-Aware NeRFs for Burst-Denoising](https://mlsanity.com/paper/2204.04668/) (2022)
- [Sparse Sampling Transformer with Uncertainty-Driven Ranking for Unified Removal of Raindrops and Rain Streaks](https://mlsanity.com/paper/2308.14153/) (2023)
- [Semi-Supervised Video Deraining with Dynamical Rain Generator](https://mlsanity.com/paper/2103.07939/) (2021)
- [Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework](https://mlsanity.com/paper/2103.12318/) (2021)
- [FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors](https://mlsanity.com/paper/1803.07487/) (2018)
- [NightRain: Nighttime Video Deraining via Adaptive-Rain-Removal and Adaptive-Correction](https://mlsanity.com/paper/2401.00729/) (2024)
- [PoRF: Pose Residual Field for Accurate Neural Surface Reconstruction](https://mlsanity.com/paper/2310.07449/) (2023)
- [Unsupervised Image Deraining: Optimization Model Driven Deep CNN](https://mlsanity.com/paper/2203.13699/) (2022)

## Built on by (most cited first, 8 of 8)

- [RealX3D: A Physically-Degraded 3D Benchmark for Multi-view Visual Restoration and Reconstruction](https://mlsanity.com/paper/2512.23437/) (2025)
- [OmniRaindrop: A Comprehensive Benchmark for Dual-Focused Raindrop Removal under Day and Night Scenes](https://mlsanity.com/paper/2510.17719/) (2025)
- [Rethinking Rainy 3D Scene Reconstruction via Perspective Transforming and Brightness Tuning](https://mlsanity.com/paper/2511.06734/) (2025)
- [NimbusGS: Unified 3D Scene Reconstruction under Hybrid Weather](https://mlsanity.com/paper/2603.27228/) (2026)
- [Enhancing Underwater Imaging with 4-D Light Fields: Dataset and Method](https://mlsanity.com/paper/2408.17339/) (2024)
- [Enhancing Underwater Light Field Images via Global Geometry-aware Diffusion Process](https://mlsanity.com/paper/2601.21179/) (2026)
- [DerainSplat: Feed-Forward Clean 3D Gaussian Splatting from Sparse Rainy Views](https://mlsanity.com/paper/2608.02191/) (2026)
- [FujinSplat: Seeing Through Smoke with RAW-Domain Gaussian Splatting](https://mlsanity.com/paper/2609.06017/) (2026)

## Abstract (arXiv)

We propose RainyScape, an unsupervised framework for reconstructing clean scenes from a collection of multi-view rainy images. RainyScape consists of two main modules: a neural rendering module and a rain-prediction module that incorporates a predictor network and a learnable latent embedding that captures the rain characteristics of the scene. Specifically, based on the spectral bias property of neural networks, we first optimize the neural rendering pipeline to obtain a low-frequency scene representation. Subsequently, we jointly optimize the two modules, driven by the proposed adaptive direction-sensitive gradient-based reconstruction loss, which encourages the network to distinguish between scene details and rain streaks, facilitating the propagation of gradients to the relevant components. Extensive experiments on both the classic neural radiance field and the recently proposed 3D Gaussian splatting demonstrate the superiority of our method in effectively eliminating rain streaks and rendering clean images, achieving state-of-the-art performance. The constructed high-quality dataset and source code will be publicly available.