# Unsupervised Self-Driving Attention Prediction via Uncertainty Mining and Knowledge Embedding

arXiv: https://arxiv.org/abs/2303.09706 · HTML: https://mlsanity.com/paper/2303.09706/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 64 references that are in the map.
- First posted on arXiv 17 Mar 2023.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving](https://mlsanity.com/paper/1904.04620/) (2019)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [Explainable Object-induced Action Decision for Autonomous Vehicles](https://mlsanity.com/paper/2003.09405/) (2020)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)

## Built on by (most cited first, 19 of 19)

- [T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Driving](https://mlsanity.com/paper/2411.18894/) (2024)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning](https://mlsanity.com/paper/2512.15153/) (2025)
- [Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation](https://mlsanity.com/paper/2501.05264/) (2025)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [VLM-Assisted Continual learning for Visual Question Answering in Self-Driving](https://mlsanity.com/paper/2502.00843/) (2025)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving](https://mlsanity.com/paper/2607.04179/) (2026)
- [Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localization](https://mlsanity.com/paper/2504.13460/) (2025)
- [From Scene to Object: Text-Guided Dual-Gaze Prediction](https://mlsanity.com/paper/2604.20191/) (2026)
- [Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation](https://mlsanity.com/paper/2606.06002/) (2026)
- [A VideoMAE-v2 Approach to Zero-Shot Traffic Accident Anticipation](https://mlsanity.com/paper/2606.09542/) (2026)
- [Towards Robust Unsupervised Attention Prediction in Autonomous Driving](https://mlsanity.com/paper/2501.15045/) (2025)
- [TRUST: Test-time Resource Utilization for Superior Trustworthiness](https://mlsanity.com/paper/2506.06048/) (2025)
- [Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2511.23304/) (2025)
- [Leveraging Metric Depth for Relative Depth Prediction](https://mlsanity.com/paper/2606.10628/) (2026)
- [SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation](https://mlsanity.com/paper/2606.15328/) (2026)
- [A DVDrive Approach for doScenes Instructed Driving Challenge](https://mlsanity.com/paper/2606.21623/) (2026)
- [Boosting Robustness for All-Weather Self-Supervised Depth Estimation in Autonomous Driving](https://mlsanity.com/paper/2607.21526/) (2026)

## Abstract (arXiv)

Predicting attention regions of interest is an important yet challenging task for self-driving systems. Existing methodologies rely on large-scale labeled traffic datasets that are labor-intensive to obtain. Besides, the huge domain gap between natural scenes and traffic scenes in current datasets also limits the potential for model training. To address these challenges, we are the first to introduce an unsupervised way to predict self-driving attention by uncertainty modeling and driving knowledge integration. Our approach's Uncertainty Mining Branch (UMB) discovers commonalities and differences from multiple generated pseudo-labels achieved from models pre-trained on natural scenes by actively measuring the uncertainty. Meanwhile, our Knowledge Embedding Block (KEB) bridges the domain gap by incorporating driving knowledge to adaptively refine the generated pseudo-labels. Quantitative and qualitative results with equivalent or even more impressive performance compared to fully-supervised state-of-the-art approaches across all three public datasets demonstrate the effectiveness of the proposed method and the potential of this direction. The code will be made publicly available.