mlsanityData to 7 Oct 2026

Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

arXiv 1902.07830 ↗First posted 21 Feb 2019cs.ROcited by 313

Not on the map. Why?

Abstract (arXiv)

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

Key facts. Cited by 313 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 58; 2023: 77; 2024: 66; 2025: 50; 2026: 26.

313 papers.

Sort:

By area:

  1. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  4. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  5. 20213D Object Detection for Autonomous Driving: A Surveycited by 106
  6. 2022Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentationcited by 87
  7. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  8. 2022PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllerscited by 84
  9. 2021FINE Samples for Learning with Noisy Labelscited by 82
  10. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  11. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  12. 2022Multi-modal Sensor Fusion for Auto Driving Perception: A Surveycited by 73
38 more
  1. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  2. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  3. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  4. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  5. 2022CENet: Toward Concise and Efficient LiDAR Semantic Segmentation for Autonomous Drivingcited by 56
  6. 2022Polar Parametrization for Vision-based Surround-View 3D Detectioncited by 53
  7. 2023Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentationcited by 53
  8. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  9. 2021RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localizationcited by 49
  10. 2021Radar-Camera Pixel Depth Association for Depth Completioncited by 49
  11. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  12. 2022Review on Panoramic Imaging and Its Applications in Scene Understandingcited by 42
  13. 2021Coarse-to-Fine Domain Adaptive Semantic Segmentation with Photometric Alignment and Category-Center Regularizationcited by 36
  14. 2024LW-DETR: A Transformer Replacement to YOLO for Real-Time Detectioncited by 36
  15. 2022Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learningcited by 35
  16. 2021Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptationcited by 34
  17. 2023WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfacescited by 34
  18. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  19. 2024Frequency-Adaptive Dilated Convolution for Semantic Segmentationcited by 34
  20. 2021An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Drivingcited by 33
  21. 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
  22. 2022StructToken : Rethinking Semantic Segmentation with Structural Priorcited by 30
  23. 2022HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detectioncited by 30
  24. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  25. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  26. 2023Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attentioncited by 29
  27. 2024EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentationcited by 27
  28. 2021Multimodal Knowledge Expansioncited by 26
  29. 2022Weakly Aligned Feature Fusion for Multimodal Object Detectioncited by 26
  30. 2021Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setupscited by 25
  31. 2021Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Drivingcited by 25
  32. 2021Rethinking of Radar's Role: A Camera-Radar Dataset and Systematic Annotator via Coordinate Alignmentcited by 23
  33. 2021UMAD: Universal Model Adaptation under Domain and Category Shiftcited by 23
  34. 2022Motion Robust High-Speed Light-Weighted Object Detection With Event Cameracited by 23
  35. 2024Adaptive Bounding Box Uncertainties via Two-Step Conformal Predictioncited by 23
  36. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  37. 2021MASS: Multi-Attentional Semantic Segmentation of LiDAR Data for Dense Top-View Understandingcited by 22
  38. 2021Exploring Event-driven Dynamic Context for Accident Scene Segmentationcited by 22

50 most cited of 313 All 313 (JSON)