# AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving

arXiv: https://arxiv.org/abs/2510.27047 · HTML: https://mlsanity.com/paper/2510.27047/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [A Review on Deep Learning Techniques Applied to Semantic Segmentation](https://mlsanity.com/paper/1704.06857/) (2017)
- [Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey](https://mlsanity.com/paper/2112.08936/) (2021)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation](https://mlsanity.com/paper/2312.03502/) (2023)
- [A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images](https://mlsanity.com/paper/1912.10230/) (2019)
- [Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving](https://mlsanity.com/paper/1810.10193/) (2018)
- [A Brief Survey and an Application of Semantic Image Segmentation for Autonomous Driving](https://mlsanity.com/paper/1808.08413/) (2018)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [A Generalized Surface Loss for Reducing the Hausdorff Distance in Medical Imaging Segmentation](https://mlsanity.com/paper/2302.03868/) (2023)
- [Exploring the Benefits of Vision Foundation Models for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2406.09896/) (2024)
- [Deep Learning based Computer Vision Methods for Complex Traffic Environments Perception: A Review](https://mlsanity.com/paper/2211.05120/) (2022)
- [Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes](https://mlsanity.com/paper/2408.12406/) (2024)
- [Annotation Free Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2403.09307/) (2024)

## Abstract (arXiv)

This paper presents the Autonomous Driving Segment Anything Model (AD-SAM), a foundation-model adaptation framework for semantic segmentation in autonomous driving. AD-SAM combines a frozen Segment Anything Model (SAM) Vision Transformer (ViT-H) encoder with a trainable ResNet-50 encoder to integrate general visual representations with multi-scale, domain-specific spatial features. Features from the two encoders are integrated through deformable convolution and channel attention, followed by a multi-stage deformable decoder for semantic prediction. Training employs a hybrid objective combining Focal, Dice, Lov'asz-Softmax, and Surface losses. Experiments on Cityscapes and Berkeley DeepDrive 100K (BDD100K) show that AD-SAM outperforms SAM, Generalized SAM (G-SAM), and DeepLabV3 under a controlled training protocol. AD-SAM achieves 76.27% mIoU on Cityscapes and 64.74% on BDD100K, exceeding DeepLabV3 by 3.45 and 5.00 percentage points, respectively, with larger gains over the SAM-based baselines. Sample-size experiments reveal dataset-dependent behavior. In particular, AD-SAM performs strongly across training sizes on Cityscapes, while its advantage on the more heterogeneous BDD100K becomes pronounced with increased training data. When trained on Cityscapes and directly evaluated on BDD100K, AD-SAM achieves the highest cross-dataset retention (84.88%) among the evaluated models. AD-SAM also converges rapidly, while precomputed frozen SAM embeddings reduce training memory requirements. These findings demonstrate the potential of combining general foundation-model representations with domain-specific multi-scale features for accurate and robust autonomous-driving semantic segmentation.