# Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling

arXiv: https://arxiv.org/abs/1904.03375 · HTML: https://mlsanity.com/paper/1904.03375/

## Key facts

- Cited by 151 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 30 of 151)

- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [Taming 3DGS: High-Quality Radiance Fields with Limited Resources](https://mlsanity.com/paper/2406.15643/) (2024)
- [Geometric Transformer for Fast and Robust Point Cloud Registration](https://mlsanity.com/paper/2202.06688/) (2022)
- [Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis](https://mlsanity.com/paper/2105.01288/) (2021)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [Sonata: Self-Supervised Learning of Reliable Point Representations](https://mlsanity.com/paper/2503.16429/) (2025)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [GeoTransformer: Fast and Robust Point Cloud Registration with Geometric Transformer](https://mlsanity.com/paper/2308.03768/) (2023)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [PointCutMix: Regularization Strategy for Point Cloud Classification](https://mlsanity.com/paper/2101.01461/) (2021)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [PRA-Net: Point Relation-Aware Network for 3D Point Cloud Analysis](https://mlsanity.com/paper/2112.04903/) (2021)
- [Contrastive Boundary Learning for Point Cloud Segmentation](https://mlsanity.com/paper/2203.05272/) (2022)
- [SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection](https://mlsanity.com/paper/2201.01976/) (2022)
- [Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling](https://mlsanity.com/paper/2107.02389/) (2021)
- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [Attention-based Point Cloud Edge Sampling](https://mlsanity.com/paper/2302.14673/) (2023)
- [PointBA: Towards Backdoor Attacks in 3D Point Cloud](https://mlsanity.com/paper/2103.16074/) (2021)
- [Geometry-Aware Self-Training for Unsupervised Domain Adaptationon Object Point Clouds](https://mlsanity.com/paper/2108.09169/) (2021)
- [Transformers in 3D Point Clouds: A Survey](https://mlsanity.com/paper/2205.07417/) (2022)
- [FESTA: Flow Estimation via Spatial-Temporal Attention for Scene Point Clouds](https://mlsanity.com/paper/2104.00798/) (2021)
- [PointConvFormer: Revenge of the Point-based Convolution](https://mlsanity.com/paper/2208.02879/) (2022)
- [SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition](https://mlsanity.com/paper/2310.07189/) (2023)
- [Weakly Supervised Semantic Segmentation for Large-Scale Point Cloud](https://mlsanity.com/paper/2212.04744/) (2022)
- [3DCTN: 3D Convolution-Transformer Network for Point Cloud Classification](https://mlsanity.com/paper/2203.00828/) (2022)
- [Isometric 3D Adversarial Examples in the Physical World](https://mlsanity.com/paper/2210.15291/) (2022)

## Abstract (arXiv)

Geometric deep learning is increasingly important thanks to the popularity of 3D sensors. Inspired by the recent advances in NLP domain, the self-attention transformer is introduced to consume the point clouds. We develop Point Attention Transformers (PATs), using a parameter-efficient Group Shuffle Attention (GSA) to replace the costly Multi-Head Attention. We demonstrate its ability to process size-varying inputs, and prove its permutation equivariance. Besides, prior work uses heuristics dependence on the input data (e.g., Furthest Point Sampling) to hierarchically select subsets of input points. Thereby, we for the first time propose an end-to-end learnable and task-agnostic sampling operation, named Gumbel Subset Sampling (GSS), to select a representative subset of input points. Equipped with Gumbel-Softmax, it produces a "soft" continuous subset in training phase, and a "hard" discrete subset in test phase. By selecting representative subsets in a hierarchical fashion, the networks learn a stronger representation of the input sets with lower computation cost. Experiments on classification and segmentation benchmarks show the effectiveness and efficiency of our methods. Furthermore, we propose a novel application, to process event camera stream as point clouds, and achieve a state-of-the-art performance on DVS128 Gesture Dataset.