# Seeing Behind Dynamic Occlusions with Event Cameras

arXiv: https://arxiv.org/abs/2307.15829 · HTML: https://mlsanity.com/paper/2307.15829/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera](https://mlsanity.com/paper/1811.10180/) (2018)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1811.08230/) (2018)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding](https://mlsanity.com/paper/2203.00867/) (2022)
- [Image Inpainting via Conditional Texture and Structure Dual Generation](https://mlsanity.com/paper/2108.09760/) (2021)
- [Learning to See Through Obstructions](https://mlsanity.com/paper/2004.01180/) (2020)
- [Multi-Bracket High Dynamic Range Imaging with Event Cameras](https://mlsanity.com/paper/2203.06622/) (2022)
- [Reduce Information Loss in Transformers for Pluralistic Image Inpainting](https://mlsanity.com/paper/2205.05076/) (2022)
- [MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting](https://mlsanity.com/paper/2203.06304/) (2022)
- [An Asynchronous Kalman Filter for Hybrid Event Cameras](https://mlsanity.com/paper/2012.05590/) (2020)
- [Event-based Synthetic Aperture Imaging with a Hybrid Network](https://mlsanity.com/paper/2103.02376/) (2021)
- [DeOccNet: Learning to See Through Foreground Occlusions in Light Fields](https://mlsanity.com/paper/1912.04459/) (2019)
- [Let's See Clearly: Contaminant Artifact Removal for Moving Cameras](https://mlsanity.com/paper/2104.08852/) (2021)
- [Learning to See Through with Events](https://mlsanity.com/paper/2212.02219/) (2022)

## Built on by (most cited first, 3 of 3)

- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [Event-Based De-Snowing for Autonomous Driving](https://mlsanity.com/paper/2507.20901/) (2025)

## Abstract (arXiv)

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, can severely degrade the performance of computer-vision systems. Dynamic occlusions are particularly challenging because of the continuously changing pattern. Existing occlusion-removal methods currently use synthetic aperture imaging or image inpainting. However, they face issues with dynamic occlusions as these require multiple viewpoints or user-generated masks to hallucinate the background intensity. We propose a novel approach to reconstruct the background from a single viewpoint in the presence of dynamic occlusions. Our solution relies for the first time on the combination of a traditional camera with an event camera. When an occlusion moves across a background image, it causes intensity changes that trigger events. These events provide additional information on the relative intensity changes between foreground and background at a high temporal resolution, enabling a truer reconstruction of the background content. We present the first large-scale dataset consisting of synchronized images and event sequences to evaluate our approach. We show that our method outperforms image inpainting methods by 3dB in terms of PSNR on our dataset.