# Instruct2See: Learning to Remove Any Obstructions Across Distributions

arXiv: https://arxiv.org/abs/2505.17649 · HTML: https://mlsanity.com/paper/2505.17649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models](https://mlsanity.com/paper/2207.14626/) (2022)
- [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017)
- [DesnowNet: Context-Aware Deep Network for Snow Removal](https://mlsanity.com/paper/1708.04512/) (2017)
- [Single Image Reflection Separation with Perceptual Losses](https://mlsanity.com/paper/1806.05376/) (2018)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [OneRestore: A Universal Restoration Framework for Composite Degradation](https://mlsanity.com/paper/2407.04621/) (2024)
- [Restoring Images in Adverse Weather Conditions via Histogram Transformer](https://mlsanity.com/paper/2407.10172/) (2024)
- [Flare7K: A Phenomenological Nighttime Flare Removal Dataset](https://mlsanity.com/paper/2210.06570/) (2022)
- [Sparse Sampling Transformer with Uncertainty-Driven Ranking for Unified Removal of Raindrops and Rain Streaks](https://mlsanity.com/paper/2308.14153/) (2023)
- [Nighttime Smartphone Reflective Flare Removal Using Optical Center Symmetry Prior](https://mlsanity.com/paper/2303.15046/) (2023)
- [Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recovery](https://mlsanity.com/paper/2308.16460/) (2023)
- [Zero-shot Object Counting with Good Exemplars](https://mlsanity.com/paper/2407.04948/) (2024)
- [Learning to See Through Obstructions with Layered Decomposition](https://mlsanity.com/paper/2008.04902/) (2020)
- [Blind Image Decomposition](https://mlsanity.com/paper/2108.11364/) (2021)
- [Gradpaint: Gradient-Guided Inpainting with Diffusion Models](https://mlsanity.com/paper/2309.09614/) (2023)
- [Neural Spline Fields for Burst Image Fusion and Layer Separation](https://mlsanity.com/paper/2312.14235/) (2023)
- [UAV-Rain1k: A Benchmark for Raindrop Removal from UAV Aerial Imagery](https://mlsanity.com/paper/2402.05773/) (2024)
- [Dual-Pixel Raindrop Removal](https://mlsanity.com/paper/2210.13321/) (2022)
- [Accurate and efficient video de-fencing using convolutional neural networks and temporal information](https://mlsanity.com/paper/1806.10781/) (2018)
- [Efficient Flow-Guided Multi-frame De-fencing](https://mlsanity.com/paper/2301.10759/) (2023)

## Built on by (most cited first, 3 of 3)

- [Unpaired Image Deraining Using Reward-Guided Self-Reinforcement Strategy](https://mlsanity.com/paper/2605.00719/) (2026)
- [DSP-Reg: Domain-Sensitive Parameter Regularization for Robust Domain Generalization](https://mlsanity.com/paper/2601.19394/) (2026)
- [Freq-DP Net: A Dual-Branch Network for Fence Removal using Dual-Pixel and Fourier Priors](https://mlsanity.com/paper/2602.14226/) (2026)

## Abstract (arXiv)

Images are often obstructed by various obstacles due to capture limitations, hindering the observation of objects of interest. Most existing methods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, making comprehensive data collection impractical. To overcome these challenges, we propose Instruct2See, a novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating it as a soft-hard mask restoration problem, where any obstruction can be represented using multi-modal prompts, such as visual semantics and textual instructions, processed through a cross-attention unit to enhance contextual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjustment of inaccurate masks. Extensive experiments on both in-distribution and out-of-distribution obstacles show that Instruct2See consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during the training phase. Code and dataset are available at https://jhscut.github.io/Instruct2See.