# 3D Vision with Transformers: A Survey

arXiv: https://arxiv.org/abs/2208.04309 · HTML: https://mlsanity.com/paper/2208.04309/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 145 of its 185 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 30 of 145)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)

## Built on by (most cited first, 21 of 21)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [Point Cloud Mamba: Point Cloud Learning via State Space Model](https://mlsanity.com/paper/2403.00762/) (2024)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention](https://mlsanity.com/paper/2403.08182/) (2024)
- [How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?](https://mlsanity.com/paper/2511.05449/) (2025)
- [RadarFormer: Lightweight and Accurate Real-Time Radar Object Detection Model](https://mlsanity.com/paper/2304.08447/) (2023)
- [AutoHood3D: A Multi-Modal Benchmark for Automotive Hood Design and Fluid-Structure Interaction](https://mlsanity.com/paper/2511.05596/) (2025)
- [Transformer-based stereo-aware 3D object detection from binocular images](https://mlsanity.com/paper/2304.11906/) (2023)
- [Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attacks](https://mlsanity.com/paper/2311.17128/) (2023)
- [Language Guided Domain Generalized Medical Image Segmentation](https://mlsanity.com/paper/2404.01272/) (2024)
- [Multi-Granularity Language-Guided Training for Multi-Object Tracking](https://mlsanity.com/paper/2406.04844/) (2024)
- [Deep Generative Models on 3D Representations: A Survey](https://mlsanity.com/paper/2210.15663/) (2022)
- [Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer](https://mlsanity.com/paper/2312.09529/) (2023)
- [HoloMine: A Synthetic Dataset for Buried Landmines Recognition using Microwave Holographic Imaging](https://mlsanity.com/paper/2502.21054/) (2025)
- [Efficient 3D Object Reconstruction using Visual Transformers](https://mlsanity.com/paper/2302.08474/) (2023)
- [Learnable Weight Initialization for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2306.09320/) (2023)
- [LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression](https://mlsanity.com/paper/2310.00369/) (2023)
- [Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training](https://mlsanity.com/paper/2409.14940/) (2024)
- [MLPerf Automotive](https://mlsanity.com/paper/2510.27065/) (2025)

## Abstract (arXiv)

The success of the transformer architecture in natural language processing has recently triggered attention in the computer vision field. The transformer has been used as a replacement for the widely used convolution operators, due to its ability to learn long-range dependencies. This replacement was proven to be successful in numerous tasks, in which several state-of-the-art methods rely on transformers for better learning. In computer vision, the 3D field has also witnessed an increase in employing the transformer for 3D convolution neural networks and multi-layer perceptron networks. Although a number of surveys have focused on transformers in vision in general, 3D vision requires special attention due to the difference in data representation and processing when compared to 2D vision. In this work, we present a systematic and thorough review of more than 100 transformers methods for different 3D vision tasks, including classification, segmentation, detection, completion, pose estimation, and others. We discuss transformer design in 3D vision, which allows it to process data with various 3D representations. For each application, we highlight key properties and contributions of proposed transformer-based methods. To assess the competitiveness of these methods, we compare their performance to common non-transformer methods on 12 3D benchmarks. We conclude the survey by discussing different open directions and challenges for transformers in 3D vision. In addition to the presented papers, we aim to frequently update the latest relevant papers along with their corresponding implementations at: https://github.com/lahoud/3d-vision-transformers.