mlsanityData to 7 Oct 2026

CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

arXiv 2608.09202 ↗First posted 10 Aug 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Modern autonomous vehicles are equipped with multiple sensors, such as cameras, LiDAR, and radar, for comprehensive environmental perception. However, robust cross-modal feature fusion remains a critical challenge, as the reliability of each sensor varies significantly across diverse real-world driving conditions, including poor visibility and adverse weather. While uncertainty quantification (UQ) mitigates this issue by allowing models to prioritize reliable signals, existing uncertainty-aware fusion methods typically rely on simple feature-level uncertainty estimates and thus often fail to generalize effectively in complex, out-of-distribution scenarios. To address this limitation, we propose CRUISE, a novel uncertainty-aware cross-modal sensor fusion framework. CRUISE integrates a vision-language model (VLM)-guided UQ module that generates fine-grained, pixel-level uncertainty estimates. By leveraging the VLM's rich prior knowledge and superior contextual reasoning, our approach provides a highly informative guide for the fusion process. Furthermore, we introduce a dynamic adaptive mechanism that explicitly models and captures cross-modal dependencies, ensuring the framework fully exploits the inherent complementary nature of multi-sensor inputs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 84 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

61 of 84 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  9. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  10. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
38 more
  1. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  3. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  4. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  5. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  6. 2017Submanifold Sparse Convolutional Networkscited by 223
  7. 2019SGD: General Analysis and Improved Ratescited by 206
  8. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  9. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  10. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  11. 2021Multimodal Virtual Point 3D Detectioncited by 166
  12. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 163
  13. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  14. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  15. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 128
  16. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 115
  17. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  18. 2020Lidar for Autonomous Driving: The principles, challenges, and trends for automotive lidar and perception systemscited by 100
  19. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  20. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  21. 2023FocalFormer3D : Focusing on Hard Instance for 3D Object Detectioncited by 58
  22. 2019Sensor Fusion for Joint 3D Object Detection and Semantic Segmentationcited by 48
  23. 2021FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentationcited by 45
  24. 2020Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understandingcited by 42
  25. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  26. 2024StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentationcited by 38
  27. 2024CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenescited by 33
  28. 2022HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detectioncited by 30
  29. 2022aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perceptioncited by 29
  30. 2019Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogatecited by 28
  31. 2022Latent Discriminant deterministic Uncertaintycited by 17
  32. 2023RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understandingcited by 17
  33. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  34. 2022HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perceptioncited by 16
  35. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalizationcited by 15
  36. 2022EcoFusion: Energy-Aware Adaptive Sensor Fusion for Efficient Autonomous Vehicle Perceptioncited by 13
  37. 2023DistHD: A Learner-Aware Dynamic Encoding Method for Hyperdimensional Classificationcited by 13
  38. 2023Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimationcited by 11

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
  2. 2026LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design
  3. 2025A Unified Perception-Language-Action Framework for Adaptive Autonomous Driving
  4. 2025SUPER-AD: Semantic Uncertainty-aware Planning for End-to-End Robust Autonomous Driving
  5. 2025PhysNav-DG: A Novel Adaptive Framework for Robust VLM-Sensor Fusion in Navigation Applications
  6. 2026UniUncer: Unified Dynamic Static Uncertainty for End to End Driving
  7. 2025VLC Fusion: Vision-Language Conditioned Sensor Fusion for Robust Object Detection
  8. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty