mlsanityData to 14 Oct 2026

Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives

arXiv 2402.02968 ↗First posted 5 Feb 2024cs.CV cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Foundation models have indeed made a profound impact on various fields, emerging as pivotal components that significantly shape the capabilities of intelligent systems. In the context of intelligent vehicles, leveraging the power of foundation models has proven to be transformative, offering notable advancements in visual understanding. Equipped with multi-modal and multi-task learning capabilities, multi-modal multi-task visual understanding foundation models (MM-VUFMs) effectively process and fuse data from diverse modalities and simultaneously handle various driving-related tasks with powerful adaptability, contributing to a more holistic understanding of the surrounding scene. In this survey, we present a systematic analysis of MM-VUFMs specifically designed for road scenes. Our objective is not only to provide a comprehensive overview of common practices, referring to task-specific models, unified multi-modal models, unified multi-task models, and foundation model prompting techniques, but also to highlight their advanced capabilities in diverse learning paradigms. These paradigms include open-world understanding, efficient transfer for road scenes, continual learning, interactive and generative capability. Moreover, we provide insights into key challenges and future trends, such as closed-loop driving systems, interpretability, embodied driving agents, and world models. To facilitate researchers in staying abreast of the latest developments in MM-VUFMs for road scenes, we have established a continuously updated repository at https://github.com/rolsheng/MM-VUFM4DS

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 224 of its 268 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

224 of 268 references.

References · 224

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
38 more
  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  13. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  14. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  15. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  16. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  17. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  18. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  19. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  20. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  21. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  22. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  23. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  24. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  25. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  26. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  27. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  28. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  29. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  30. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  31. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  32. 2016Progressive Neural Networkscited by 1,361
  33. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  34. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  35. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  36. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  37. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  38. 2022Planning-oriented Autonomous Drivingcited by 1,087

50 most cited of 224 All 224 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testingcited by 10
  3. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  4. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  5. 2024V-RoAst: Visual Road Assessment. Can VLM be a Road Safety Assessor Using the iRAP Standard?cited by 4
  6. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  7. 2025OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedbackcited by 4
  8. 2024TiCoSS: Tightening the Coupling between Semantic Segmentation and Stereo Matching within A Joint Learning Frameworkcited by 3
  9. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  10. 2025Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MRcited by 3
  11. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  12. 2025Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understandingcited by 3
11 more
  1. 2025Multi-label Scene Classification for Autonomous Vehicles: Acquiring and Accumulating Knowledge from Diverse Datasetscited by 2
  2. 2025Reinforcement Learning for Adaptive Planner Parameter Tuning: A Perspective on Hierarchical Architecturecited by 1
  3. 2026Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localizationcited by 1
  4. 2024TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuningcited by 0
  5. 2025Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulationcited by 0
  6. 2025Universal Incremental Learning: Mitigating Confusion from Inter- and Intra-task Distribution Randomnesscited by 0
  7. 2025A Survey on Deep Multi-Task Learning in Connected Autonomous Vehiclescited by 0
  8. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0
  9. 2026Behavior-Grounded Lane Representation Learning for Multi-Task Traffic Digital Twinscited by 0
  10. 2026TextDS: Parameter-Efficient Representation Alignment for Scene Text Detection under Distribution Shiftscited by 0
  11. 2026Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraintcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models
  2. 2025Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving
  3. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
  4. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
  5. 2024Probing Multimodal LLMs as World Models for Driving
  6. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
  7. 2025RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding
  8. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving