# After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data

arXiv: https://arxiv.org/abs/2609.38607 · HTML: https://mlsanity.com/paper/2609.38607/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 30)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022)
- [From Shadow Generation to Shadow Removal](https://mlsanity.com/paper/2103.12997/) (2021)
- [Physics-based Shadow Image Decomposition for Shadow Removal](https://mlsanity.com/paper/2012.13018/) (2020)
- [Instance Shadow Detection](https://mlsanity.com/paper/1911.07034/) (2019)
- [High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net](https://mlsanity.com/paper/2308.14221/) (2023)
- [Revisiting Shadow Detection: A New Benchmark Dataset for Complex World](https://mlsanity.com/paper/1911.06998/) (2019)
- [Learning from Synthetic Shadows for Shadow Detection and Removal](https://mlsanity.com/paper/2101.01713/) (2021)
- [Leveraging Inpainting for Single-Image Shadow Removal](https://mlsanity.com/paper/2302.05361/) (2023)
- [Triple-cooperative Video Shadow Detection](https://mlsanity.com/paper/2103.06533/) (2021)
- [SILT: Shadow-aware Iterative Label Tuning for Learning to Detect Shadows from Noisy Labels](https://mlsanity.com/paper/2308.12064/) (2023)
- [OmniSR: Shadow Removal under Direct and Indirect Lighting](https://mlsanity.com/paper/2410.01719/) (2024)
- [Detail-Preserving Latent Diffusion for Stable Shadow Removal](https://mlsanity.com/paper/2412.17630/) (2024)
- [Intrinsic Decomposition of Document Images In-the-Wild](https://mlsanity.com/paper/2011.14447/) (2020)
- [Shadow Removal Refinement via Material-Consistent Shadow Edges](https://mlsanity.com/paper/2409.06848/) (2024)
- [S-EO: A Large-Scale Dataset for Geometry-Aware Shadow Detection in Remote Sensing Applications](https://mlsanity.com/paper/2504.06920/) (2025)
- [PhaSR: Generalized Image Shadow Removal with Physically Aligned Priors](https://mlsanity.com/paper/2601.17470/) (2026)
- [Gen-n-Val: Agentic Image Data Generation and Validation](https://mlsanity.com/paper/2506.04676/) (2025)
- [Beyond Shadows: A Large-Scale Benchmark and Multi-Stage Framework for High-Fidelity Facial Shadow Removal](https://mlsanity.com/paper/2601.19309/) (2026)
- [See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis](https://mlsanity.com/paper/2602.20951/) (2026)

## Abstract (arXiv)

Shadow removal looks nearly solved on established benchmarks, yet remains brittle in the real world. Models have advanced; the paired training data they rely on have barely changed in nearly a decade. The reason is simple: obtaining a shadow-free target requires removing the occluder while keeping the scene, camera, and illumination otherwise unchanged, making diverse paired data difficult to capture. Meanwhile, large shadow detection datasets already contain diverse real-world images and masks, but no shadow-free targets. To turn this abundant but incomplete data into paired supervision, we propose an offline agentic workflow combining physics-motivated generation, failure detection, feedback-driven retry, candidate selection, and deterministic correction. Using this workflow, we construct AgenticShadow, a dataset of 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing. Our construction workflow reduces Color Distribution Difference by 50.5% over previous shadow removal work, while training existing shadow removal models on AgenticShadow reduces cross-domain LAB RMSE by 19.7-37.5%.