# WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models

arXiv: https://arxiv.org/abs/2305.07528 · HTML: https://mlsanity.com/paper/2305.07528/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 60 references that are in the map.
- First posted on arXiv 12 May 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Semantic Foggy Scene Understanding with Synthetic Data](https://mlsanity.com/paper/1708.07819/) (2017)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Scale-Aware Trident Networks for Object Detection](https://mlsanity.com/paper/1901.01892/) (2019)
- [Imbalance Problems in Object Detection: A Review](https://mlsanity.com/paper/1909.00169/) (2019)
- [Deep MANTA: A Coarse-to-fine Many-Task Network for joint 2D and 3D vehicle analysis from monocular image](https://mlsanity.com/paper/1703.07570/) (2017)
- [Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2](https://mlsanity.com/paper/2210.00586/) (2022)
- [Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network](https://mlsanity.com/paper/1803.04271/) (2018)
- [DAWN: Vehicle Detection in Adverse Weather Nature Dataset](https://mlsanity.com/paper/2008.05402/) (2020)

## Built on by (most cited first, 28 of 28)

- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning](https://mlsanity.com/paper/2402.12715/) (2024)
- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Reliability in Semantic Segmentation: Can We Use Synthetic Data?](https://mlsanity.com/paper/2312.09231/) (2023)
- [Generative Models in Decision Making: A Survey](https://mlsanity.com/paper/2502.17100/) (2025)
- [Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models](https://mlsanity.com/paper/2310.08577/) (2023)
- [SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection](https://mlsanity.com/paper/2411.05633/) (2024)
- [Prospective Role of Foundation Models in Advancing Autonomous Vehicles](https://mlsanity.com/paper/2405.02288/) (2023)
- [Real-Time Environment Condition Classification for Autonomous Vehicles](https://mlsanity.com/paper/2405.19305/) (2024)
- [Identifying Systematic Errors in Object Detectors with the SCROD Pipeline](https://mlsanity.com/paper/2309.13489/) (2023)
- [Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models](https://mlsanity.com/paper/2405.20991/) (2024)
- [A Survey on Large Language Model-empowered Autonomous Driving](https://mlsanity.com/paper/2409.14165/) (2024)
- [GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models](https://mlsanity.com/paper/2307.12018/) (2023)
- [How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perception](https://mlsanity.com/paper/2408.17222/) (2024)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL](https://mlsanity.com/paper/2511.12755/) (2025)
- [t-RAIN: Robust generalization under weather-aliasing label shift attacks](https://mlsanity.com/paper/2305.08302/) (2023)
- [Reinforcement Learning from Diffusion Feedback: Q* for Image Search](https://mlsanity.com/paper/2311.15648/) (2023)
- [GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis](https://mlsanity.com/paper/2401.15282/) (2024)
- [Identification of Fine-grained Systematic Errors via Controlled Scene Generation](https://mlsanity.com/paper/2404.07045/) (2024)
- [Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations](https://mlsanity.com/paper/2408.17311/) (2024)
- [The SET Perceptual Factors Framework: Towards Assured Perception for Autonomous Systems](https://mlsanity.com/paper/2508.10798/) (2025)
- [Localization-Guided Foreground Augmentation in Autonomous Driving](https://mlsanity.com/paper/2604.18940/) (2026)
- [Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Models](https://mlsanity.com/paper/2605.07649/) (2026)
- [XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions](https://mlsanity.com/paper/2605.11521/) (2026)
- [Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support](https://mlsanity.com/paper/2606.00991/) (2026)
- [What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs](https://mlsanity.com/paper/2606.01624/) (2026)
- [Weather-Aware Domain Adaptation for Street-View Weather Recognition](https://mlsanity.com/paper/2610.02000/) (2026)

## Abstract (arXiv)

The open road poses many challenges to autonomous perception, including poor visibility from extreme weather conditions. Models trained on good-weather datasets frequently fail at detection in these out-of-distribution settings. To aid adversarial robustness in perception, we introduce WEDGE (WEather images by DALL-E GEneration): a synthetic dataset generated with a vision-language generative model via prompting. WEDGE consists of 3360 images in 16 extreme weather conditions manually annotated with 16513 bounding boxes, supporting research in the tasks of weather classification and 2D object detection. We have analyzed WEDGE from research standpoints, verifying its effectiveness for extreme-weather autonomous perception. We establish baseline performance for classification and detection with 53.87% test accuracy and 45.41 mAP. Most importantly, WEDGE can be used to fine-tune state-of-the-art detectors, improving SOTA performance on real-world weather benchmarks (such as DAWN) by 4.48 AP for well-generated classes like trucks. WEDGE has been collected under OpenAI's terms of use and is released for public use under the CC BY-NC-SA 4.0 license. The repository for this work and dataset is available at https://infernolia.github.io/WEDGE.