mlsanityData to 7 Oct 2026

Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsing

arXiv 2502.06219 ↗First posted 10 Feb 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent vision foundation models (VFMs), typically based on Vision Transformer (ViT), have significantly advanced numerous computer vision tasks. Despite their success in tasks focused solely on RGB images, the potential of VFMs in RGB-depth driving scene parsing remains largely under-explored. In this article, we take one step toward this emerging research area by investigating a feasible technique to fully exploit VFMs for generalizable RGB-depth driving scene parsing. Specifically, we explore the inherent characteristics of RGB and depth data, thereby presenting a Heterogeneous Feature Integration Transformer (HFIT). This network enables the efficient extraction and integration of comprehensive heterogeneous features without re-training ViTs. Relative depth prediction results from VFMs, used as inputs to the HFIT side adapter, overcome the limitations of the dependence on depth maps. Our proposed HFIT demonstrates superior performance compared to all other traditional single-modal and data-fusion scene parsing networks, pre-trained VFMs, and ViT adapters on the Cityscapes and KITTI Semantics datasets. We believe this novel strategy paves the way for future innovations in VFM-based data-fusion techniques for driving scene parsing. Our source code is publicly available at https://mias.group/HFIT.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

50 of 65 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2016Layer Normalizationcited by 5,579
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2024Depth Anything V2cited by 1,385
  11. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  12. 2019UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentationcited by 583
38 more
  1. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  2. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  4. 2021RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matchingcited by 348
  5. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  6. 2021K-Net: Towards Unified Image Segmentationcited by 211
  7. 2019Expectation-Maximization Attention Networks for Semantic Segmentationcited by 147
  8. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 120
  9. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  10. 2014Salient Object Detection: A Discriminative Regional Feature Integration Approachcited by 89
  11. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 83
  12. 2021FaPN: Feature-aligned Pyramid Network for Dense Image Predictioncited by 81
  13. 2020Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detectioncited by 72
  14. 2020SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detectioncited by 66
  15. 2020Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimationcited by 66
  16. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 65
  17. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  18. 2019GFF: Gated Fully Fusion for Semantic Segmentationcited by 57
  19. 2021Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusioncited by 56
  20. 2019Pothole Detection Based on Disparity Transformation and Road Surface Modelingcited by 38
  21. 2020Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Imagescited by 36
  22. 2022ORFD: A Dataset and Benchmark for Off-Road Freespace Detectioncited by 33
  23. 2021Learning to Relate Depth and Semantics for Unsupervised Domain Adaptationcited by 32
  24. 2023RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsingcited by 27
  25. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 26
  26. 2016Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networkscited by 24
  27. 2024Playing to Vision Foundation Model's Strengths in Stereo Matchingcited by 23
  28. 2021Graph Attention Layer Evolves Semantic Segmentation for Road Pothole Detection: A Benchmark and Algorithmscited by 15
  29. 2024S3M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Drivingcited by 14
  30. 2022Robust RGB-D Fusion for Saliency Detectioncited by 12
  31. 2024Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matchingcited by 12
  32. 2024HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusioncited by 10
  33. 2024RoadFormer+: Delivering RGB-X Scene Parsing through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusioncited by 9
  34. 2023Joint Depth Prediction and Semantic Segmentation with Multi-View SAMcited by 8
  35. 2024LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Drivingcited by 8
  36. 2024ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Predictioncited by 8
  37. 2024These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusioncited by 5
  38. 2025Environment-Driven Online LiDAR-Camera Extrinsic Calibrationcited by 3

Built on by

Not cited yet.