# Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality

arXiv: https://arxiv.org/abs/2412.16481 · HTML: https://mlsanity.com/paper/2412.16481/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 21 Dec 2024.

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking](https://mlsanity.com/paper/2109.03805/) (2021)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [Online normalizer calculation for softmax](https://mlsanity.com/paper/1805.02867/) (2018)
- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation](https://mlsanity.com/paper/2008.01550/) (2020)
- [OctFormer: Octree-based Transformers for 3D Point Clouds](https://mlsanity.com/paper/2305.03045/) (2023)
- [SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2210.07372/) (2022)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer](https://mlsanity.com/paper/2301.08739/) (2023)
- [Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion](https://mlsanity.com/paper/2103.07074/) (2021)
- [FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving](https://mlsanity.com/paper/2308.01006/) (2023)
- [ThunderKittens: Simple, Fast, and Adorable AI Kernels](https://mlsanity.com/paper/2410.20399/) (2024)
- [BuildingNet: Learning to Label 3D Buildings](https://mlsanity.com/paper/2110.04955/) (2021)
- [Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks](https://mlsanity.com/paper/1906.08989/) (2019)

## Built on by (most cited first, 1 of 1)

- [LitePT: Lighter Yet Stronger Point Transformer](https://mlsanity.com/paper/2512.13689/) (2025)

## Abstract (arXiv)

Recent efforts recognize the power of scale in 3D learning (e.g. PTv3) and attention mechanisms (e.g. FlashAttention). However, current point cloud backbones fail to holistically unify geometric locality, attention mechanisms, and GPU architectures in one view. In this paper, we introduce Flash3D Transformer, which aligns geometric locality and GPU tiling through a principled locality mechanism based on Perfect Spatial Hashing (PSH). The common alignment with GPU tiling naturally fuses our PSH locality mechanism with FlashAttention at negligible extra cost. This mechanism affords flexible design choices throughout the backbone that result in superior downstream task results. Flash3D outperforms state-of-the-art PTv3 results on benchmark datasets, delivering a 2.25x speed increase and 2.4x memory efficiency boost. This efficiency enables scaling to wider attention scopes and larger models without additional overhead. Such scaling allows Flash3D to achieve even higher task accuracies than PTv3 under the same compute budget.