# PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection

arXiv: https://arxiv.org/abs/2312.08371 · HTML: https://mlsanity.com/paper/2312.08371/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 35 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [3D Object Detection with Pointformer](https://mlsanity.com/paper/2012.11409/) (2020)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds](https://mlsanity.com/paper/2203.11139/) (2022)
- [LiDAR R-CNN: An Efficient and Universal 3D Object Detector](https://mlsanity.com/paper/2103.15297/) (2021)
- [RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection](https://mlsanity.com/paper/2106.13365/) (2021)
- [MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer](https://mlsanity.com/paper/2203.10981/) (2022)
- [Improving 3D Object Detection with Channel-wise Transformer](https://mlsanity.com/paper/2108.10723/) (2021)
- [PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection](https://mlsanity.com/paper/2205.07403/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2210.07372/) (2022)
- [Fully Sparse 3D Object Detection](https://mlsanity.com/paper/2207.10035/) (2022)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [3D-MAN: 3D Multi-frame Attention Network for Object Detection](https://mlsanity.com/paper/2103.16054/) (2021)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [M3DSSD: Monocular 3D Single Stage Object Detector](https://mlsanity.com/paper/2103.13164/) (2021)
- [MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2303.08316/) (2023)

## Built on by (most cited first, 8 of 8)

- [V2X-R: Cooperative LiDAR-4D Radar Fusion with Denoising Diffusion for 3D Object Detection](https://mlsanity.com/paper/2411.08402/) (2024)
- [Image-Guided Outdoor LiDAR Perception Quality Assessment for Autonomous Driving](https://mlsanity.com/paper/2406.17265/) (2024)
- [Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences](https://mlsanity.com/paper/2409.04390/) (2024)
- [Towards a 3D Transfer-based Black-box Attack via Critical Feature Guidance](https://mlsanity.com/paper/2508.15650/) (2025)
- [FASTer: Focal Token Acquiring-and-Scaling Transformer for Long-term 3D Object Detection](https://mlsanity.com/paper/2503.01899/) (2025)
- [SSCATeR: Sparse Scatter-Based Convolution Algorithm with Temporal Data Recycling for Real-Time 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/2512.08557/) (2025)
- [Scene Reconstruction as Mapping Priors for 3D Detection](https://mlsanity.com/paper/2605.22997/) (2026)
- [ATLAS: A Large-Scale Evaluation Benchmark for Adversarial LiDAR Perception](https://mlsanity.com/paper/2606.02924/) (2026)

## Abstract (arXiv)

Recent temporal LiDAR-based 3D object detectors achieve promising performance based on the two-stage proposal-based approach. They generate 3D box candidates from the first-stage dense detector, followed by different temporal aggregation methods. However, these approaches require per-frame objects or whole point clouds, posing challenges related to memory bank utilization. Moreover, point clouds and trajectory features are combined solely based on concatenation, which may neglect effective interactions between them. In this paper, we propose a point-trajectory transformer with long short-term memory for efficient temporal 3D object detection. To this end, we only utilize point clouds of current-frame objects and their historical trajectories as input to minimize the memory bank storage requirement. Furthermore, we introduce modules to encode trajectory features, focusing on long short-term and future-aware perspectives, and then effectively aggregate them with point cloud features. We conduct extensive experiments on the large-scale Waymo dataset to demonstrate that our approach performs well against state-of-the-art methods. Code and models will be made publicly available at https://github.com/kuanchihhuang/PTT.