mlsanityData to 7 Oct 2026

Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection

arXiv 2310.15670 ↗First posted 24 Oct 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Current research is primarily dedicated to advancing the accuracy of camera-only 3D object detectors (apprentice) through the knowledge transferred from LiDAR- or multi-modal-based counterparts (expert). However, the presence of the domain gap between LiDAR and camera features, coupled with the inherent incompatibility in temporal fusion, significantly hinders the effectiveness of distillation-based enhancements for apprentices. Motivated by the success of uni-modal distillation, an apprentice-friendly expert model would predominantly rely on camera features, while still achieving comparable performance to multi-modal models. To this end, we introduce VCD, a framework to improve the camera-only apprentice model, including an apprentice-friendly multi-modal expert and temporal-fusion-friendly distillation supervision. The multi-modal expert VCD-E adopts an identical structure as that of the camera-only apprentice in order to alleviate the feature disparity, and leverages LiDAR input as a depth prior to reconstruct the 3D scene, achieving the performance on par with other heterogeneous multi-modal experts. Additionally, a fine-grained trajectory-based distillation module is introduced with the purpose of individually rectifying the motion misalignment for each object in the scene. With those improvements, our camera-only apprentice VCD-A sets new state-of-the-art on nuScenes with a score of 63.1% NDS.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 57 of its 63 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

57 of 63 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2016Layer Normalizationcited by 5,579
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2019Objects as Pointscited by 1,172
  11. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
38 more
  1. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  2. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  3. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  4. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  6. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  7. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  8. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  9. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  10. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  11. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  12. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  13. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  14. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  15. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  16. 2023Scene as Occupancycited by 208
  17. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  18. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  19. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  20. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  21. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  22. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  23. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  24. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  25. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  26. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  27. 2022BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereocited by 89
  28. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  29. 2021Monocular 3D Object Detection: An Extrinsic Parameter Free Approachcited by 69
  30. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 69
  31. 2022MonoDistill: Learning Spatial Features for Monocular 3D Object Detectioncited by 68
  32. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  33. 2023Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perceptioncited by 65
  34. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  35. 2022STS: Surround-view Temporal Stereo for Multi-view 3D Detectioncited by 53
  36. 2023CAPE: Camera View Position Embedding for Multi-View 3D Object Detectioncited by 39
  37. 2022TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learningcited by 37
  38. 2022Reversible Column Networkscited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 4; 2025: 5.

13 papers.

Sort:

By area:

  1. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  2. 2024Embodied Understanding of Driving Scenarioscited by 63
  3. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  4. 2023BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detectioncited by 35
  5. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  6. 2025SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weathercited by 10
  7. 2024LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detectioncited by 9
  8. 2024TiGDistill-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning Distillationcited by 4
  9. 2025GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Modelscited by 3
  10. 2025RCTDistill: Cross-Modal Knowledge Distillation Framework for Radar-Camera 3D Object Detection with Temporal Fusioncited by 3
  11. 2024NeRF-DetS: Enhanced Adaptive Spatial-wise Sampling and View-wise Fusion Strategies for NeRF-based Indoor Multi-view 3D Object Detectioncited by 2
  12. 2025LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptationcited by 1
1 more
  1. 2025DySS: Dynamic Queries and State-Space Learning for Efficient 3D Object Detection from Multi-Camera Videoscited by 0