# OcTr: Octree-based Transformer for 3D Object Detection

arXiv: https://arxiv.org/abs/2303.12621 · HTML: https://mlsanity.com/paper/2303.12621/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 66 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)

## Built on by (most cited first, 30 of 32)

- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries](https://mlsanity.com/paper/2312.03774/) (2023)
- [HEDNet: A Hierarchical Encoder-Decoder Network for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2310.20234/) (2023)
- [LION: Linear Group RNN for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2407.18232/) (2024)
- [SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detection](https://mlsanity.com/paper/2307.11477/) (2023)
- [Vision-based 3D occupancy prediction in autonomous driving: a review and outlook](https://mlsanity.com/paper/2405.02595/) (2024)
- [AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer](https://mlsanity.com/paper/2407.12951/) (2024)
- [CoSDH: Communication-Efficient Collaborative Perception via Supply-Demand Awareness and Intermediate-Late Hybridization](https://mlsanity.com/paper/2503.03430/) (2025)
- [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024)
- [APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers](https://mlsanity.com/paper/2504.02508/) (2025)
- [FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation](https://mlsanity.com/paper/2506.11543/) (2025)
- [ScatterFormer: Efficient Voxel Transformer with Scattered Linear Attention](https://mlsanity.com/paper/2401.00912/) (2024)
- [UniMamba: Unified Spatial-Channel Representation Learning with Group-Efficient Mamba for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2503.12009/) (2025)
- [HOTFormerLoc: Hierarchical Octree Transformer for Versatile Lidar Place Recognition Across Ground and Aerial Views](https://mlsanity.com/paper/2503.08140/) (2025)
- [Unified Cross-Scale 3D Generation and Understanding via Autoregressive Modeling](https://mlsanity.com/paper/2503.16278/) (2025)
- [STAL3D: Unsupervised Domain Adaptation for 3D Object Detection via Collaborating Self-Training and Adversarial Learning](https://mlsanity.com/paper/2406.19362/) (2024)
- [FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection](https://mlsanity.com/paper/2407.10135/) (2024)
- [FusionViT: Hierarchical 3D Object Detection via LiDAR-Camera Vision Transformer Fusion](https://mlsanity.com/paper/2311.03620/) (2023)
- [Multistream Network for LiDAR and Camera-based 3D Object Detection in Outdoor Scenes](https://mlsanity.com/paper/2507.19304/) (2025)
- [Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes](https://mlsanity.com/paper/2508.16812/) (2025)
- [Leveraging Anchor-based LiDAR 3D Object Detection via Point Assisted Sample Selection](https://mlsanity.com/paper/2403.01978/) (2024)
- [IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers](https://mlsanity.com/paper/2511.15369/) (2025)
- [SDVRF: Sparse-to-Dense Voxel Region Fusion for Multi-modal 3D Object Detection](https://mlsanity.com/paper/2304.08304/) (2023)
- [R2Det: Redemption from Range-view for Accurate 3D Object Detection](https://mlsanity.com/paper/2307.11482/) (2023)
- [Decoupled and Interactive Regression Modeling for High-performance One-stage 3D Object Detection](https://mlsanity.com/paper/2409.00690/) (2024)
- [An Investigation of Beam Density on LiDAR Object Detection Performance](https://mlsanity.com/paper/2503.15087/) (2025)
- [FALO: Fast and Accurate LiDAR 3D Object Detection on Resource-Constrained Devices](https://mlsanity.com/paper/2506.04499/) (2025)
- [Voxel Densification for Serialized 3D Object Detection: Mitigating Sparsity via Pre-serialization Expansion](https://mlsanity.com/paper/2508.16069/) (2025)
- [TransBridge: Boost 3D Object Detection by Scene-Level Completion with Transformer Decoder](https://mlsanity.com/paper/2512.11926/) (2025)
- [RayMamba: Ray-Aligned Serialization for Long-Range 3D Object Detection](https://mlsanity.com/paper/2604.02903/) (2026)

## Abstract (arXiv)

A key challenge for LiDAR-based 3D object detection is to capture sufficient features from large scale 3D scenes especially for distant or/and occluded objects. Albeit recent efforts made by Transformers with the long sequence modeling capability, they fail to properly balance the accuracy and efficiency, suffering from inadequate receptive fields or coarse-grained holistic correlations. In this paper, we propose an Octree-based Transformer, named OcTr, to address this issue. It first constructs a dynamic octree on the hierarchical feature pyramid through conducting self-attention on the top level and then recursively propagates to the level below restricted by the octants, which captures rich global context in a coarse-to-fine manner while maintaining the computational complexity under control. Furthermore, for enhanced foreground perception, we propose a hybrid positional embedding, composed of the semantic-aware positional embedding and attention mask, to fully exploit semantic and geometry clues. Extensive experiments are conducted on the Waymo Open Dataset and KITTI Dataset, and OcTr reaches newly state-of-the-art results.