# PVT: Point-Voxel Transformer for Point Cloud Learning

arXiv: https://arxiv.org/abs/2108.06076 · HTML: https://mlsanity.com/paper/2108.06076/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 68 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [SO-Net: Self-Organizing Network for Point Cloud Analysis](https://mlsanity.com/paper/1803.04249/) (2018)

## Built on by (most cited first, 30 of 43)

- [Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Clouds](https://mlsanity.com/paper/2207.00531/) (2022)
- [Smooth, exact rotational symmetrization for deep learning on point clouds](https://mlsanity.com/paper/2305.19302/) (2023)
- [Deep Learning-based 3D Point Cloud Classification: A Systematic Survey and Outlook](https://mlsanity.com/paper/2311.02608/) (2023)
- [CASSPR: Cross Attention Single Scan Place Recognition](https://mlsanity.com/paper/2211.12542/) (2022)
- [Point Cloud Classification Using Content-based Transformer via Clustering in Feature Space](https://mlsanity.com/paper/2303.04599/) (2023)
- [Human-centric Scene Understanding for 3D Large-scale Scenarios](https://mlsanity.com/paper/2307.14392/) (2023)
- [Gaussian Radar Transformer for Semantic Segmentation in Noisy Radar Data](https://mlsanity.com/paper/2212.03690/) (2022)
- [Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes](https://mlsanity.com/paper/2312.04043/) (2023)
- [A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation](https://mlsanity.com/paper/2405.11903/) (2024)
- [GPSFormer: A Global Perception and Local Structure Fitting-based Transformer for Point Cloud Understanding](https://mlsanity.com/paper/2407.13519/) (2024)
- [TSegFormer: 3D Tooth Segmentation in Intraoral Scans with Geometry Guided Transformer](https://mlsanity.com/paper/2311.13234/) (2023)
- [3DHacker: Spectrum-based Decision Boundary Generation for Hard-label 3D Point Cloud Attack](https://mlsanity.com/paper/2308.07546/) (2023)
- [Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Clouds](https://mlsanity.com/paper/2309.16435/) (2023)
- [Region-Enhanced Feature Learning for Scene Semantic Segmentation](https://mlsanity.com/paper/2304.07486/) (2023)
- [PoissonNet: A Local-Global Approach for Learning on Surfaces](https://mlsanity.com/paper/2510.14146/) (2025)
- [LACV-Net: Semantic Segmentation of Large-Scale Point Cloud Scene via Local Adaptive and Comprehensive VLAD](https://mlsanity.com/paper/2210.05870/) (2022)
- [CurveCloudNet: Processing Point Clouds with 1D Structure](https://mlsanity.com/paper/2303.12050/) (2023)
- [Introducing the Short-Time Fourier Kolmogorov Arnold Network: A Dynamic Graph CNN Approach for Tree Species Classification in 3D Point Clouds](https://mlsanity.com/paper/2503.23647/) (2025)
- [SAT: Size-Aware Transformer for 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2301.06869/) (2023)
- [GTNet: Graph Transformer Network for 3D Point Cloud Classification and Semantic Segmentation](https://mlsanity.com/paper/2305.15213/) (2023)
- [ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers](https://mlsanity.com/paper/2306.10798/) (2023)
- [GeoAuxNet: Towards Universal 3D Representation Learning for Multi-sensor Point Clouds](https://mlsanity.com/paper/2403.19220/) (2024)
- [General Rotation Invariance Learning for Point Clouds via Weight-Feature Alignment](https://mlsanity.com/paper/2302.09907/) (2023)
- [Dynamic Clustering Transformer Network for Point Cloud Segmentation](https://mlsanity.com/paper/2306.08073/) (2023)
- [GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning](https://mlsanity.com/paper/2409.04963/) (2024)
- [Radar Tracker: Moving Instance Tracking in Sparse and Noisy Radar Point Clouds](https://mlsanity.com/paper/2507.03441/) (2025)
- [TFormer: 3D Tooth Segmentation in Mesh Scans with Geometry Guided Transformer](https://mlsanity.com/paper/2210.16627/) (2022)
- [Sampled Transformer for Point Sets](https://mlsanity.com/paper/2302.14346/) (2023)
- [Learned Gridification for Efficient Point Cloud Processing](https://mlsanity.com/paper/2307.14354/) (2023)
- [VoxelKP: A Voxel-based Network Architecture for Human Keypoint Estimation in LiDAR Data](https://mlsanity.com/paper/2312.08871/) (2023)

## Abstract (arXiv)

The recently developed pure Transformer architectures have attained promising accuracy on point cloud learning benchmarks compared to convolutional neural networks. However, existing point cloud Transformers are computationally expensive since they waste a significant amount of time on structuring the irregular data. To solve this shortcoming, we present Sparse Window Attention (SWA) module to gather coarse-grained local features from non-empty voxels, which not only bypasses the expensive irregular data structuring and invalid empty voxel computation, but also obtains linear computational complexity with respect to voxel resolution. Meanwhile, to gather fine-grained features about the global shape, we introduce relative attention (RA) module, a more robust self-attention variant for rigid transformations of objects. Equipped with the SWA and RA, we construct our neural architecture called PVT that integrates both modules into a joint framework for point cloud learning. Compared with previous Transformer-based and attention-based models, our method attains top accuracy of 94.0% on classification benchmark and 10x inference speedup on average. Extensive experiments also valid the effectiveness of PVT on part and semantic segmentation benchmarks (86.6% and 69.2% mIoU, respectively).