mlsanityData to 7 Oct 2026

CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection

arXiv 2204.00325 ↗First posted 1 Apr 2022cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

In autonomous driving, LiDAR point-clouds and RGB images are two major data modalities with complementary cues for 3D object detection. However, it is quite difficult to sufficiently use them, due to large inter-modal discrepancies. To address this issue, we propose a novel framework, namely Contrastively Augmented Transformer for multi-modal 3D object Detection (CAT-Det). Specifically, CAT-Det adopts a two-stream structure consisting of a Pointformer (PT) branch, an Imageformer (IT) branch along with a Cross-Modal Transformer (CMT) module. PT, IT and CMT jointly encode intra-modal and inter-modal long-range contexts for representing an object, thus fully exploring multi-modal information for detection. Furthermore, we propose an effective One-way Multi-modal Data Augmentation (OMDA) approach via hierarchical contrastive learning at both the point and object levels, significantly improving the accuracy only by augmenting point-clouds, which is free from complex generation of paired samples of the two modalities. Extensive experiments on the KITTI benchmark show that CAT-Det achieves a new state-of-the-art, highlighting its effectiveness.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 53 of its 65 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

53 of 65 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  9. 2020Point Transformercited by 1,462
  10. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  12. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
38 more
  1. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  2. 2020PCT: Point cloud transformercited by 730
  3. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  4. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  5. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  6. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  7. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  8. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 366
  9. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  10. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  11. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 325
  12. 2019STD: Sparse-to-Dense 3D Object Detector for Point Cloudcited by 282
  13. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 277
  14. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  15. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  16. 2021Voxel Transformer for 3D Object Detectioncited by 247
  17. 2021Group-Free 3D Object Detection via Transformerscited by 215
  18. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  19. 20203D Object Detection with Pointformercited by 194
  20. 2019Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Drivingcited by 188
  21. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  22. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  23. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  24. 2019Stereo R-CNN based 3D Object Detection for Autonomous Drivingcited by 163
  25. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  26. 2019Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detectioncited by 124
  27. 2021SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloudcited by 119
  28. 2019TANet: Robust 3D Object Detection from Point Clouds with Triple Attentioncited by 108
  29. 2021Improving 3D Object Detection with Channel-wise Transformercited by 108
  30. 2019GS3D: An Efficient 3D Object Detection Framework for Autonomous Drivingcited by 106
  31. 2020DSGN: Deep Stereo Geometry Network for 3D Object Detectioncited by 100
  32. 2017Deep MANTA: A Coarse-to-fine Many-Task Network for joint 2D and 3D vehicle analysis from monocular imagecited by 95
  33. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  34. 2019Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspotscited by 84
  35. 2021M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformerscited by 74
  36. 2016Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networkscited by 72
  37. 2019PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Modulecited by 56
  38. 2020P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understandingcited by 41

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 16; 2024: 2; 2025: 4; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2023Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Drivingcited by 112
  4. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  5. 2023RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Drivingcited by 76
  6. 2022MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessmentcited by 38
  7. 2023GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detectioncited by 34
  8. 2024GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detectioncited by 33
  9. 2023OcTr: Octree-based Transformer for 3D Object Detectioncited by 32
  10. 2022Transformers in 3D Point Clouds: A Surveycited by 31
  11. 2023SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detectioncited by 30
  12. 20223D Vision with Transformers: A Surveycited by 21
20 more
  1. 2022ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditionscited by 21
  2. 2023Unleash the Potential of Image Branch for Cross-modal 3D Object Detectioncited by 11
  3. 2023Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videoscited by 8
  4. 2025Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Camerascited by 8
  5. 2023MLF-DET: Multi-Level Fusion for Cross-Modal 3D Object Detectioncited by 5
  6. 2023InteractionNet: Joint Planning and Prediction for Autonomous Driving with Transformerscited by 4
  7. 2022PV-RCNN++: Semantical Point-Voxel Feature Interaction for 3D Object Detectioncited by 3
  8. 2023Efficient Transformer-based 3D Object Detection with Dynamic Token Haltingcited by 3
  9. 2023FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detectioncited by 1
  10. 2023What You See Is What You Detect: Towards better Object Densification in 3D detectioncited by 1
  11. 2025DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detectioncited by 1
  12. 2022Recursive Cross-View: Use Only 2D Detectors to Achieve 3D Object Detection without 3D Annotationscited by 0
  13. 2022ImLiDAR: Cross-Sensor Dynamic Message Propagation Network for 3D Object Detectioncited by 0
  14. 2023PTA-Det: Point Transformer Associating Point cloud and Image for 3D Object Detectioncited by 0
  15. 2023R2Det: Redemption from Range-view for Accurate 3D Object Detectioncited by 0
  16. 2023Dissecting RGB-D Learning for Improved Multi-modal Fusioncited by 0
  17. 2024Image-Guided Semantic Pseudo-LiDAR Point Generation for 3D Object Detectioncited by 0
  18. 2025Efficient Multimodal 3D Object Detector via Instance-Level Contrastive Distillationcited by 0
  19. 2025Revisiting Radar Camera Alignment by Contrastive Learning for 3D Object Detectioncited by 0
  20. 2026Embodied Semantic Communication for Collective Autonomous Agents: A Tutorial on Representation, Wireless Delivery, and Closed-Loop Coordinationcited by 0