# Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case

arXiv: https://arxiv.org/abs/2608.06075 · HTML: https://mlsanity.com/paper/2608.06075/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 13 of 13)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [From Shadow Segmentation to Shadow Removal](https://mlsanity.com/paper/2008.00267/) (2020)
- [Leveraging Inpainting for Single-Image Shadow Removal](https://mlsanity.com/paper/2302.05361/) (2023)
- [Shadow Removal Refinement via Material-Consistent Shadow Edges](https://mlsanity.com/paper/2409.06848/) (2024)

## Built on by (most cited first, 1 of 1)

- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)

## Abstract (arXiv)

Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision? We study this through shadow removal, a problem shaped by scene geometry, illumination, materials, and occluders, where paired shadow and shadow-free data are hard to collect at scale. We find that a commercial generative editor, used directly, can produce clean shadow-free edits that preserve surface texture and local appearance. However, this comes with a new failure mode: the same editor can regenerate scene content, hallucinate objects, or misread a shadow as material or geometry, producing plausible but physically wrong edits. We address this with an agentic candidate-selection pipeline: the editor generates a guided probe, an evaluator screens for major failures, retries when needed, samples multiple candidates, filters them, and selects a final result balancing shadow removal against scene preservation. Grounding this process in shadow-formation physics makes it more reliable: prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion, not material or object structure, measurably improves quality and consistency. On the ShadowRemovalRefine benchmark, our physics-oriented pipeline achieves a CDD of 0.0075, reducing CDD by at least 47% over the strongest prior method. These results suggest that commercial vision-language models do not replace classic low-level vision priors; instead, such priors remain useful for constraining and steering physically underconstrained generation.