# IntrinsicWeather: Controllable Weather Editing in Intrinsic Space

arXiv: https://arxiv.org/abs/2508.06982 · HTML: https://mlsanity.com/paper/2508.06982/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 9 Aug 2025.

## Builds on (most cited first, 30 of 37)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions](https://mlsanity.com/paper/2111.14813/) (2021)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond](https://mlsanity.com/paper/2309.16553/) (2023)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [Intrinsic Image Diffusion for Indoor Single-view Material Estimation](https://mlsanity.com/paper/2312.12274/) (2023)
- [Restoring Images in Adverse Weather Conditions via Histogram Transformer](https://mlsanity.com/paper/2407.10172/) (2024)
- [DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models](https://mlsanity.com/paper/2501.18590/) (2025)
- [IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations](https://mlsanity.com/paper/2412.12083/) (2024)

## Built on by (most cited first, 2 of 2)

- [AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing](https://mlsanity.com/paper/2603.26546/) (2026)
- [MeteoVerse: Unified Weather-Controllable Video World Model](https://mlsanity.com/paper/2609.36810/) (2026)

## Abstract (arXiv)

We present IntrinsicWeather, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches. We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. IntrinsicWeather outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.