# FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling

arXiv: https://arxiv.org/abs/2511.12708 · HTML: https://mlsanity.com/paper/2511.12708/

## Key facts

- Cited by 2 papers in the map (data through 7 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Multi-scale Interactive Network for Salient Object Detection](https://mlsanity.com/paper/2007.09062/) (2020)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.08052/) (2025)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)

## Built on by (most cited first, 2 of 2)

- [From Scene to Object: Text-Guided Dual-Gaze Prediction](https://mlsanity.com/paper/2604.20191/) (2026)
- [FocusDrive: Reasoning with Visual Focus for Autonomous Driving](https://mlsanity.com/paper/2609.33190/) (2026)

## Abstract (arXiv)

Understanding not only where drivers look but also why their attention shifts is essential for interpretable human-AI collaboration in autonomous driving. Driver attention is not purely perceptual but semantically structured. Thus, attention shifts can be learned through minimal semantic supervision rather than dense large-scale annotation. We present FSDAM (Few-Shot Driver Attention Modeling), a framework that achieves joint spatial attention prediction and structured explanation generation using 90 annotated examples. Our key insight is to decompose attention into an explicit reasoning representation, including scene context, current focus, anticipated next focus, and causal explanation, and to learn next-focus anticipation through minimal-pair supervision. To address task conflict and large sample requirements of existing models, and to mitigate task interference under limited data, we introduce a novel dual-pathway architecture in which separate modules handle spatial prediction and caption generation. In addition, we use a training-only vision-language alignment mechanism that injects semantic priors into spatial learning without increasing inference complexity, mitigating task interference under few-shot training. Despite extreme data scarcity, FSDAM achieves competitive performance in gaze prediction, and generates coherent, context-aware structural reasoning for improved interpretability. The model further demonstrates strong zero-shot generalization across multiple driving benchmarks.