# Distilling Temporal Knowledge with Masked Feature Reconstruction for 3D Object Detection

arXiv: https://arxiv.org/abs/2401.01918 · HTML: https://mlsanity.com/paper/2401.01918/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision](https://mlsanity.com/paper/2211.10439/) (2022)
- [Distilling Object Detectors with Fine-grained Feature Imitation](https://mlsanity.com/paper/1906.03609/) (2019)
- [Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection](https://mlsanity.com/paper/2210.02443/) (2022)
- [Channel-wise Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/2011.13256/) (2020)
- [Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion](https://mlsanity.com/paper/2211.10581/) (2022)
- [SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos](https://mlsanity.com/paper/2308.09244/) (2023)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [General Instance Distillation for Object Detection](https://mlsanity.com/paper/2103.02340/) (2021)
- [Masked Generative Distillation](https://mlsanity.com/paper/2205.01529/) (2022)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Sparse4D v2: Recurrent Temporal Fusion with Sparse Model](https://mlsanity.com/paper/2305.14018/) (2023)
- [Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perception](https://mlsanity.com/paper/2303.05970/) (2023)
- [UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View](https://mlsanity.com/paper/2303.15083/) (2023)
- [DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2309.15109/) (2023)
- [Masked Distillation with Receptive Tokens](https://mlsanity.com/paper/2205.14589/) (2022)

## Abstract (arXiv)

Striking a balance between precision and efficiency presents a prominent challenge in the bird's-eye-view (BEV) 3D object detection. Although previous camera-based BEV methods achieved remarkable performance by incorporating long-term temporal information, most of them still face the problem of low efficiency. One potential solution is knowledge distillation. Existing distillation methods only focus on reconstructing spatial features, while overlooking temporal knowledge. To this end, we propose TempDistiller, a Temporal knowledge Distiller, to acquire long-term memory from a teacher detector when provided with a limited number of frames. Specifically, a reconstruction target is formulated by integrating long-term temporal knowledge through self-attention operation applied to feature teachers. Subsequently, novel features are generated for masked student features via a generator. Ultimately, we utilize this reconstruction target to reconstruct the student features. In addition, we also explore temporal relational knowledge when inputting full frames for the student model. We verify the effectiveness of the proposed method on the nuScenes benchmark. The experimental results show our method obtain an enhancement of +1.6 mAP and +1.1 NDS compared to the baseline, a speed improvement of approximately 6 FPS after compressing temporal knowledge, and the most accurate velocity estimation.