mlsanityData to 7 Oct 2026

An End-to-End Transformer Model for 3D Object Detection

arXiv 2109.08141 ↗First posted 16 Sep 2021cs.CV cs.AI cs.LGcited by 329

Not on the map. Why?

Abstract (arXiv)

We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds. Compared to existing detection methods that employ a number of 3D-specific inductive biases, 3DETR requires minimal modifications to the vanilla Transformer block. Specifically, we find that a standard Transformer with non-parametric queries and Fourier positional embeddings is competitive with specialized architectures that employ libraries of 3D-specific operators with hand-tuned hyperparameters. Nevertheless, 3DETR is conceptually simple and easy to implement, enabling further improvements by incorporating 3D domain knowledge. Through extensive experiments, we show 3DETR outperforms the well-established and highly optimized VoteNet baselines on the challenging ScanNetV2 dataset by 9.5%. Furthermore, we show 3DETR is applicable to 3D tasks beyond detection, and can serve as a building block for future research.

Key facts. Cited by 329 papers in the map (data through 7 Oct 2026). Builds on 71 of its 93 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

71 of 93 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2016Layer Normalizationcited by 5,579
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2020Point Transformercited by 1,462
  8. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  9. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  12. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  13. 2018Image Transformercited by 770
  14. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  15. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  16. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  17. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  18. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  19. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  20. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  21. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  22. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  23. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  24. 20203DSSD: Point-based 3D Single Stage Object Detectorcited by 440
  25. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  26. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  27. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  28. 2017Relation Networks for Object Detectioncited by 305
  29. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  30. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  31. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  32. 20203D Object Detection with Pointformercited by 194
  33. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  34. 2018Tangent Convolutions for Dense Prediction in 3Dcited by 173
  35. 2018SPLATNet: Sparse Lattice Networks for Point Cloud Processingcited by 158
  36. 2019Modeling Point Clouds with Self-Attention and Gumbel Subset Samplingcited by 151
  37. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  38. 2015End-to-end people detection in crowded scenescited by 140

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 95; 2023: 97; 2024: 63; 2025: 42; 2026: 23.

329 papers.

Sort:

By area:

  1. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  2. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  3. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  4. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  5. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  6. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  7. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  8. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  9. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  10. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  11. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  12. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
38 more
  1. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  2. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  4. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  5. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  6. 2022DETRs with Hybrid Matchingcited by 149
  7. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  10. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  11. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  12. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  13. 2021A Survey of Visual Transformerscited by 117
  14. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  15. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  16. 2022Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Cloudscited by 102
  17. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  18. 2022CenterFormer: Center-based Transformer for 3D Object Detectioncited by 99
  19. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  20. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  21. 2022REGTR: End-to-end Point Cloud Correspondences with Transformerscited by 94
  22. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  23. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  24. 2022MonoDETR: Depth-guided Transformer for Monocular 3D Object Detectioncited by 85
  25. 2022Surface Representation for Point Cloudscited by 84
  26. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
  27. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  28. 2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
  29. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  30. 2022MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstructioncited by 72
  31. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  32. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  33. 2022Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstructioncited by 67
  34. 2022Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imagingcited by 66
  35. 2022CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformercited by 66
  36. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  37. 2021Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinderscited by 59
  38. 2022Flow-Guided Transformer for Video Inpaintingcited by 59

50 most cited of 329 All 329 (JSON)