# Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving

arXiv: https://arxiv.org/abs/2011.13628 · HTML: https://mlsanity.com/paper/2011.13628/

## Key facts

- Cited by 55 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 30 of 55)

- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception](https://mlsanity.com/paper/2307.13929/) (2023)
- [Transformers in 3D Point Clouds: A Survey](https://mlsanity.com/paper/2205.07417/) (2022)
- [SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2403.07284/) (2024)
- [From Multi-View to Hollow-3D: Hallucinated Hollow-3D R-CNN for 3D Object Detection](https://mlsanity.com/paper/2107.14391/) (2021)
- [Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.04448/) (2022)
- [GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation](https://mlsanity.com/paper/2206.06420/) (2022)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trends](https://mlsanity.com/paper/2402.15490/) (2024)
- [MGTANet: Encoding Sequential LiDAR Points Using Long Short-Term Motion-Guided Temporal Attention for 3D Object Detection](https://mlsanity.com/paper/2212.00442/) (2022)
- [TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses](https://mlsanity.com/paper/2306.05888/) (2023)
- [SVQNet: Sparse Voxel-Adjacent Query Network for 4D Spatio-Temporal LiDAR Semantic Segmentation](https://mlsanity.com/paper/2308.13323/) (2023)
- [DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions](https://mlsanity.com/paper/2412.10739/) (2024)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Fine-Grained Image Style Transfer with Visual Transformers](https://mlsanity.com/paper/2210.05176/) (2022)
- [A Survey of Robust 3D Object Detection Methods in Point Clouds](https://mlsanity.com/paper/2204.00106/) (2022)
- [SSD-MonoDETR: Supervised Scale-aware Deformable Transformer for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.07270/) (2023)
- [Text-Driven Traffic Anomaly Detection with Temporal High-Frequency Modeling in Driving Videos](https://mlsanity.com/paper/2401.03522/) (2024)
- [X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation](https://mlsanity.com/paper/2409.19342/) (2024)
- [TransPillars: Coarse-to-Fine Aggregation for Multi-Frame 3D Object Detection](https://mlsanity.com/paper/2208.03141/) (2022)
- [CoDynTrust: Robust Asynchronous Collaborative Perception via Dynamic Feature Trust Modulus](https://mlsanity.com/paper/2502.08169/) (2025)
- [Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving](https://mlsanity.com/paper/2103.08116/) (2021)
- [Transformers Meet Visual Learning Understanding: A Comprehensive Review](https://mlsanity.com/paper/2203.12944/) (2022)

## Abstract (arXiv)

The strong demand of autonomous driving in the industry has lead to strong interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal information of the sequence of data. In this work, we propose a new transformer, called Temporal-Channel Transformer, to model the spatial-temporal domain and channel domain relationships for video object detecting from Lidar data. As a special design of this transformer, the information encoded in the encoder is different from that in the decoder, i.e. the encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-channel information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer will decode the information for each location of the current frame. Before conducting the object detection with detection head, the gate mechanism is deployed for re-calibrating the features of current frame, which filters out the object irrelevant information by repetitively refine the representation of target frame along with the up-sampling process. Experimental results show that we achieve the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark.