mlsanityData to 7 Oct 2026

Cross Modal Transformer: Towards Fast and Robust 3D Object Detection

arXiv 2301.01283 ↗First posted 3 Jan 2023cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. It achieves 74.1% NDS (state-of-the-art with single model) on nuScenes test set while maintaining fast inference speed. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code is released at https://github.com/junjie18/CMT.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 53 of its 63 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

53 of 63 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  9. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2020Center-based 3D Object Detection and Trackingcited by 1,143
38 more
  1. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  2. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  3. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  4. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  5. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  6. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  7. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  8. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  10. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  11. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  12. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  13. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  14. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  15. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  16. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  17. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  18. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  19. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  20. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  21. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  22. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  23. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  24. 2019CenterMask : Real-Time Anchor-Free Instance Segmentationcited by 200
  25. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  26. 2021Probabilistic and Geometric Depth: Detecting Objects in Perspectivecited by 190
  27. 2021Instances as Queriescited by 157
  28. 2022DETRs with Hybrid Matchingcited by 149
  29. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  30. 2021RangeDet:In Defense of Range View for LiDAR-based 3D Object Detectioncited by 140
  31. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  32. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  33. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  34. 2021RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detectioncited by 115
  35. 2021FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detectioncited by 92
  36. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  37. 2022LargeKernel3D: Scaling up Kernels in 3D Sparse CNNscited by 71
  38. 2021SOLQ: Segmenting Objects by Learning Queriescited by 68

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 21; 2026: 19.

68 papers.

Sort:

By area:

  1. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  2. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  3. 2024Ray Denoising: Depth-aware Hard Negative Sampling for Multi-view 3D Object Detectioncited by 36
  4. 2023BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detectioncited by 35
  5. 2024Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perceptioncited by 35
  6. 2024SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detectioncited by 25
  7. 2024MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detectioncited by 23
  8. 2023HVDetFusion: A Simple and Robust Camera-Radar Fusion Frameworkcited by 18
  9. 2023CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementaritycited by 18
  10. 2024DeepInteraction++: Multi-Modality Interaction for Autonomous Drivingcited by 15
  11. 2025MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perceptioncited by 15
  12. 2024ContrastAlign: Toward Robust BEV Feature Alignment via Contrastive Learning for Multi-Modal 3D Object Detectioncited by 13
38 more
  1. 2025CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perceptioncited by 13
  2. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  3. 2024GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Drivingcited by 11
  4. 2023Long-Tailed 3D Detection via Multi-Modal Fusioncited by 10
  5. 2024Progressive Multi-Modal Fusion for Robust 3D Object Detectioncited by 10
  6. 2025SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weathercited by 10
  7. 2024FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Drivingcited by 9
  8. 2024SparseFusion: Efficient Sparse Multi-Modal Fusion Framework for Long-Range 3D Perceptioncited by 8
  9. 2024SimpleBEV: Improved LiDAR-Camera Fusion Architecture for 3D Object Detectioncited by 8
  10. 2024SpaRC: Sparse Radar-Camera Fusion for 3D Object Detectioncited by 8
  11. 2025How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentationcited by 7
  12. 2024Towards Unified 3D Object Detection via Algorithm and Data Unificationcited by 6
  13. 2025Availability-aware Sensor Fusion via Unified Canonical Spacecited by 5
  14. 2026UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentationcited by 5
  15. 2024Image-Guided Outdoor LiDAR Perception Quality Assessment for Autonomous Drivingcited by 3
  16. 2024BiCo-Fusion: Bidirectional Complementary LiDAR-Camera Fusion for Semantic- and Spatial-Aware 3D Object Detectioncited by 3
  17. 2024Multi-modal Crowd Counting via Modal Emulationcited by 3
  18. 2025Recent Advances in Transformer and Large Language Models for UAV Applicationscited by 3
  19. 2025UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNscited by 3
  20. 2026Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detectioncited by 3
  21. 2025CoreNet: Conflict Resolution Network for Point-Pixel Misalignment and Sub-Task Suppression of 3D LiDAR-Camera Object Detectioncited by 2
  22. 2025CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Predictioncited by 2
  23. 2025CRAB: Camera-Radar Fusion for Reducing Depth Ambiguity in Backward Projection based View Transformationcited by 2
  24. 2025Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognitioncited by 2
  25. 2026To View Transform or Not to View Transform: NeRF-based Pre-training Perspectivecited by 2
  26. 2023Learned Fusion: 3D Object Detection using Calibration-Free Transformer Feature Fusioncited by 1
  27. 2023Explainable Multi-Camera 3D Object Detection with Transformer-Based Saliency Mapscited by 1
  28. 2024FADet: A Multi-sensor 3D Object Detection Network based on Local Featured Attentioncited by 1
  29. 2024AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstructioncited by 1
  30. 2024Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representationcited by 1
  31. 2025LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptationcited by 1
  32. 2026FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detectioncited by 1
  33. 2024Explore the LiDAR-Camera Dynamic Adjustment Fusion for 3D Object Detectioncited by 0
  34. 2025FASTer: Focal Token Acquiring-and-Scaling Transformer for Long-term 3D Object Detectioncited by 0
  35. 2025Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detectioncited by 0
  36. 2025An Efficient LiDAR-Camera Fusion Network for Multi-Class 3D Dynamic Object Detection and Trajectory Predictioncited by 0
  37. 2025Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancycited by 0
  38. 2025CrossRay3D: Geometry and Distribution Guidance for Efficient Multimodal 3D Detectioncited by 0

50 most cited of 68 All 68 (JSON)