# PatchFormer: An Efficient Point Transformer with Patch Attention

arXiv: https://arxiv.org/abs/2111.00207 · HTML: https://mlsanity.com/paper/2111.00207/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 30 Oct 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)

## Built on by (most cited first, 30 of 43)

- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [OcTr: Octree-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2303.12621/) (2023)
- [Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registration](https://mlsanity.com/paper/2303.13290/) (2023)
- [PointVector: A Vector Representation In Point Cloud Analysis](https://mlsanity.com/paper/2205.10528/) (2022)
- [LCPFormer: Towards Effective 3D Point Cloud Analysis via Local Context Propagation in Transformers](https://mlsanity.com/paper/2210.12755/) (2022)
- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [Rethinking Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.00592/) (2024)
- [LitePT: Lighter Yet Stronger Point Transformer](https://mlsanity.com/paper/2512.13689/) (2025)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Overlap-guided Gaussian Mixture Models for Point Cloud Registration](https://mlsanity.com/paper/2210.09836/) (2022)
- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Improving transferability of 3D adversarial attacks with scale and shear transformations](https://mlsanity.com/paper/2211.01093/) (2022)
- [GMA3D: Local-Global Attention Learning to Estimate Occluded Motions of Scene Flow](https://mlsanity.com/paper/2210.03296/) (2022)
- [PIDS: Joint Point Interaction-Dimension Search for 3D Point Cloud](https://mlsanity.com/paper/2211.15759/) (2022)
- [SAT: Size-Aware Transformer for 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2301.06869/) (2023)
- [GTNet: Graph Transformer Network for 3D Point Cloud Classification and Semantic Segmentation](https://mlsanity.com/paper/2305.15213/) (2023)
- [Vision Backbone Enhancement via Multi-Stage Cross-Scale Attention](https://mlsanity.com/paper/2308.05872/) (2023)
- [Fully-Geometric Cross-Attention for Point Cloud Registration](https://mlsanity.com/paper/2502.08285/) (2025)
- [MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention](https://mlsanity.com/paper/2512.01738/) (2025)
- [Applying Plain Transformers to Real-World Point Clouds](https://mlsanity.com/paper/2303.00086/) (2023)
- [Exploiting Inductive Bias in Transformer for Point Cloud Classification and Segmentation](https://mlsanity.com/paper/2304.14124/) (2023)
- [Rethinking Attention Module Design for Point Cloud Analysis](https://mlsanity.com/paper/2407.19294/) (2024)
- [HydraMamba: Multi-Head State Space Model for Global Point Cloud Learning](https://mlsanity.com/paper/2507.19778/) (2025)
- [A Dual-Cycled Cross-View Transformer Network for Unified Road Layout Estimation and 3D Object Detection in the Bird's-Eye-View](https://mlsanity.com/paper/2209.08844/) (2022)
- [ReBotNet: Fast Real-time Video Enhancement](https://mlsanity.com/paper/2303.13504/) (2023)
- [Cloud-RAIN: Point Cloud Analysis with Reflectional Invariance](https://mlsanity.com/paper/2305.07814/) (2023)
- [Fast Kernel Scene Flow](https://mlsanity.com/paper/2403.05896/) (2024)
- [3D Learnable Supertoken Transformer for LiDAR Point Cloud Scene Segmentation](https://mlsanity.com/paper/2405.15826/) (2024)
- [SPRig: Self-Supervised Pose-Invariant Rigging from Mesh Sequences](https://mlsanity.com/paper/2602.12740/) (2026)

## Abstract (arXiv)

The point cloud learning community witnesses a modeling shift from CNNs to Transformers, where pure Transformer architectures have achieved top accuracy on the major learning benchmarks. However, existing point Transformers are computationally expensive since they need to generate a large attention map, which has quadratic complexity (both in space and time) with respect to input size. To solve this shortcoming, we introduce Patch ATtention (PAT) to adaptively learn a much smaller set of bases upon which the attention maps are computed. By a weighted summation upon these bases, PAT not only captures the global shape context but also achieves linear complexity to input size. In addition, we propose a lightweight Multi-Scale aTtention (MST) block to build attentions among features of different scales, providing the model with multi-scale features. Equipped with the PAT and MST, we construct our neural architecture called PatchFormer that integrates both modules into a joint framework for point cloud learning. Extensive experiments demonstrate that our network achieves comparable accuracy on general point cloud learning tasks with 9.2x speed-up than previous point Transformers.