mlsanityData to 14 Oct 2026

DeepInteraction: 3D Object Detection via Modality Interaction

arXiv 2208.11112 ↗First posted 23 Aug 2022cs.CVcited by 125

Not on the map. Why?

Abstract (arXiv)

Existing top-performance 3D object detectors typically rely on the multi-modal fusion strategy. This design is however fundamentally restricted due to overlooking the modality-specific useful information and finally hampering the model performance. To address this limitation, in this work we introduce a novel modality interaction strategy where individual per-modality representations are learned and maintained throughout for enabling their unique characteristics to be exploited during object detection. To realize this proposed strategy, we design a DeepInteraction architecture characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments on the large-scale nuScenes dataset show that our proposed method surpasses all prior arts often by a large margin. Crucially, our method is ranked at the first position at the highly competitive nuScenes object detection leaderboard.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 23 Aug 2022.

Builds on

41 of 60 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2017Mask R-CNNcited by 7,277
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  8. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  11. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  12. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
29 more
  1. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  2. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 658
  3. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  4. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  5. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  6. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  7. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  8. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  9. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  10. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  11. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  12. 2021Voxel Transformer for 3D Object Detectioncited by 247
  13. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  14. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  15. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  16. 2021Multimodal Virtual Point 3D Detectioncited by 166
  17. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  18. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  19. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  20. 2021RangeDet:In Defense of Range View for LiDAR-based 3D Object Detectioncited by 140
  21. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  22. 2018In Defense of Classical Image Processing: Fast Depth Completion on the CPUcited by 96
  23. 2020Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detectioncited by 78
  24. 2022AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detectioncited by 70
  25. 2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
  26. 2022Voxel Field Fusion for 3D Object Detectioncited by 50
  27. 20214D-Net for Learned Multi-Modal Alignmentcited by 44
  28. 2022Learning Ego 3D Representation as Ray Tracingcited by 43
  29. 2021To the Point: Efficient 3D Object Detection in the Range Image with Graph Convolution Kernelscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 38; 2024: 37; 2025: 23; 2026: 21.

125 papers.

Sort:

By area:

  1. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  2. 2024An Extensible Framework for Open Heterogeneous Collaborative Perceptioncited by 87
  3. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
  4. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 75
  5. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  6. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  7. 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 70
  8. 2022MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detectioncited by 62
  9. 2023FocalFormer3D : Focusing on Hard Instance for 3D Object Detectioncited by 58
  10. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  11. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  12. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
38 more
  1. 2023EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detectioncited by 38
  2. 2023BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detectioncited by 35
  3. 2024Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perceptioncited by 35
  4. 2023GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detectioncited by 34
  5. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  6. 2024GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detectioncited by 33
  7. 2023BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregationcited by 31
  8. 2023Fully Sparse Fusion for 3D Object Detectioncited by 29
  9. 2023UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalitiescited by 29
  10. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  11. 2023Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detectioncited by 28
  12. 2024CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillationcited by 28
  13. 2024Multi-Sensor Object Anomaly Detection: Unifying Appearance, Geometry, and Internal Propertiescited by 26
  14. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  15. 2024SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detectioncited by 25
  16. 2024MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detectioncited by 23
  17. 2023Language-Guided 3D Object Detection in Point Cloud for Autonomous Drivingcited by 20
  18. 2024RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAMcited by 20
  19. 2023HVDetFusion: A Simple and Robust Camera-Radar Fusion Frameworkcited by 18
  20. 2024DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentationcited by 18
  21. 2024MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detectioncited by 17
  22. 2024DeepInteraction++: Multi-Modality Interaction for Autonomous Drivingcited by 15
  23. 2024GAFusion: Adaptive Fusing LiDAR and Camera with Multiple Guidance for 3D Object Detectioncited by 15
  24. 2025MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perceptioncited by 15
  25. 2023Transformer-Based Sensor Fusion for Autonomous Driving: A Surveycited by 14
  26. 2023FusionFormer: A Multi-sensory Fusion in Bird's-Eye-View and Temporal Consistent Transformer for 3D Object Detectioncited by 14
  27. 2023PointSSC: A Cooperative Vehicle-Infrastructure Point Cloud Benchmark for Semantic Scene Completioncited by 14
  28. 2024VoxelNextFusion: A Simple, Unified and Effective Voxel Fusion Framework for Multi-Modal 3D Object Detectioncited by 14
  29. 2024MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Drivingcited by 14
  30. 2024HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detectioncited by 14
  31. 2025Resilient Sensor Fusion under Adverse Sensor Failures via Multi-Modal Expert Fusioncited by 14
  32. 2023Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detectioncited by 13
  33. 2024ContrastAlign: Toward Robust BEV Feature Alignment via Contrastive Learning for Multi-Modal 3D Object Detectioncited by 13
  34. 2023UniM2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Drivingcited by 12
  35. 2025SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weathercited by 12
  36. 2022Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detectioncited by 11
  37. 2023Fusion is Not Enough: Single Modal Attacks on Fusion Models for 3D Object Detectioncited by 11
  38. 2024Progressive Multi-Modal Fusion for Robust 3D Object Detectioncited by 11

50 most cited of 125 All 125 (JSON)