mlsanityData to 7 Oct 2026

OctFormer: Octree-based Transformers for 3D Point Clouds

arXiv 2305.03045 ↗First posted 4 May 2023cs.CV cs.GRcited by 105

Not on the map. Why?

Abstract (arXiv)

We propose octree-based transformers, named OctFormer, for 3D point cloud learning. OctFormer can not only serve as a general and effective backbone for 3D point cloud segmentation and object detection but also have linear complexity and is scalable for large-scale point clouds. The key challenge in applying transformers to point clouds is reducing the quadratic, thus overwhelming, computation complexity of attentions. To combat this issue, several works divide point clouds into non-overlapping windows and constrain attentions in each local window. However, the point number in each window varies greatly, impeding the efficient execution on GPU. Observing that attentions are robust to the shapes of local windows, we propose a novel octree attention, which leverages sorted shuffled keys of octrees to partition point clouds into local windows containing a fixed number of points while permitting shapes of windows to change freely. And we also introduce dilated octree attention to expand the receptive field further. Our octree attention can be implemented in 10 lines of code with open-sourced libraries and runs 17 times faster than other point cloud attentions when the point number exceeds 200k. Built upon the octree attention, OctFormer can be easily scaled up and achieves state-of-the-art performances on a series of 3D segmentation and detection benchmarks, surpassing previous sparse-voxel-based CNNs and point cloud transformers in terms of both efficiency and effectiveness. Notably, on the challenging ScanNet200 dataset, OctFormer outperforms sparse-voxel-based CNNs by 7.3 in mIoU. Our code and trained models are available at https://wang-ps.github.io/octformer.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 69 of its 86 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

69 of 86 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2016Layer Normalizationcited by 5,579
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2020Supervised Contrastive Learningcited by 2,803
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2020Point Transformercited by 1,462
  5. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  7. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  8. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  9. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  10. 2020PCT: Point cloud transformercited by 730
  11. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  13. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  14. 2019Panoptic Feature Pyramid Networkscited by 644
  15. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  16. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  17. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  18. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  19. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  20. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  21. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  22. 2021Early Convolutions Help Transformers See Bettercited by 395
  23. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  24. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  25. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  26. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  27. 2017Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphscited by 307
  28. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  29. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  30. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  31. 2021Voxel Transformer for 3D Object Detectioncited by 247
  32. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  33. 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
  34. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  35. 2021Group-Free 3D Object Detection via Transformerscited by 215
  36. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  37. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  38. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 35; 2025: 37; 2026: 25.

105 papers.

Sort:

By area:

  1. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  2. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  3. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  4. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  5. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  6. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  7. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  8. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66
  9. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  10. 2024OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentationcited by 43
  11. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  12. 2025DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentationcited by 31
38 more
  1. 2024TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splattingcited by 28
  2. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  3. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generationcited by 27
  4. 2024PoinTramba: A Hybrid Transformer-Mamba Framework for Point Cloud Analysiscited by 26
  5. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  6. 2024RadarPillars: Efficient Object Detection from 4D Radar Point Cloudscited by 23
  7. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Modelscited by 22
  8. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 21
  9. 2025LitePT: Lighter Yet Stronger Point Transformercited by 17
  10. 2023ConDaFormer: Disassembled Transformer with Local Structure Enhancement for 3D Point Cloud Understandingcited by 16
  11. 2024KPConvX: Modernizing Kernel Point Convolution with Kernel Attentioncited by 16
  12. 2024A Unified Framework for 3D Scene Understandingcited by 15
  13. 2024SplatFormer: Point Transformer for Robust 3D Gaussian Splattingcited by 15
  14. 2024Scalable 3D Panoptic Segmentation As Superpoint Graph Clusteringcited by 14
  15. 2023Decoupled Local Aggregation for Point Cloud Learningcited by 12
  16. 2025Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenizationcited by 12
  17. 2024Zero-Shot Multi-Object Scene Completioncited by 9
  18. 2025SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillationcited by 9
  19. 2025CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligencecited by 9
  20. 2024Exploring the design space of deep-learning-based weather forecasting systemscited by 8
  21. 2024PerLA: Perceptive 3D Language Assistantcited by 8
  22. 2025HOTFormerLoc: Hierarchical Octree Transformer for Versatile Lidar Place Recognition Across Ground and Aerial Viewscited by 8
  23. 2025Multi-Scale Neighborhood Occupancy Masked Autoencoder for Self-Supervised Learning in LiDAR Point Cloudscited by 7
  24. 2024Swin3D++: Effective Multi-Source Pretraining for 3D Indoor Scene Understandingcited by 6
  25. 2025UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splattingcited by 6
  26. 2024TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Modelscited by 5
  27. 2024Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchangecited by 5
  28. 2025BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysiscited by 5
  29. 2025Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modelingcited by 5
  30. 2026Nexus: Native Mesh Generation with Diffusioncited by 5
  31. 2024Serialized Point Mamba: A Serialized Point Cloud Mamba Segmentation Modelcited by 4
  32. 2025Mitigating Ambiguities in 3D Classification with Gaussian Splattingcited by 4
  33. 2025On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentationcited by 4
  34. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  35. 2026Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understandingcited by 4
  36. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  37. 2024SP2T: Sparse Proxy Attention for Dual-stream Point Transformercited by 3
  38. 2025Autoregressive Generation of Static and Growing Treescited by 3

50 most cited of 105 All 105 (JSON)