# DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention

arXiv: https://arxiv.org/abs/2212.07849 · HTML: https://mlsanity.com/paper/2212.07849/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 50 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)

## Built on by (most cited first, 21 of 21)

- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [FB-BEV: BEV Representation from Forward-Backward View Transformations](https://mlsanity.com/paper/2308.02236/) (2023)
- [Online Map Vectorization for Autonomous Driving: A Rasterization Perspective](https://mlsanity.com/paper/2306.10502/) (2023)
- [FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View](https://mlsanity.com/paper/2403.02710/) (2024)
- [BEVHeight++: Toward Robust Visual Centric 3D Object Detection](https://mlsanity.com/paper/2309.16179/) (2023)
- [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024)
- [SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects](https://mlsanity.com/paper/2403.20318/) (2024)
- [OCBEV: Object-Centric BEV Transformer for Multi-View 3D Object Detection](https://mlsanity.com/paper/2306.01738/) (2023)
- [SimPB: A Single Model for 2D and 3D Object Detection from Multiple Cameras](https://mlsanity.com/paper/2403.10353/) (2024)
- [DV-3DLane: End-to-end Multi-modal 3D Lane Detection with Dual-view Representation](https://mlsanity.com/paper/2406.16072/) (2024)
- [Learning High-resolution Vector Representation from Multi-Camera Images for 3D Object Detection](https://mlsanity.com/paper/2407.15354/) (2024)
- [SparseLaneSTP: Leveraging Spatio-Temporal Priors with Sparse Transformers for 3D Lane Detection](https://mlsanity.com/paper/2601.04968/) (2026)
- [EVT: Efficient View Transformation for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2411.10715/) (2024)
- [Bridging Text and Video Generation: A Survey](https://mlsanity.com/paper/2510.04999/) (2025)
- [AdaOcc: Adaptive-Resolution Occupancy Prediction](https://mlsanity.com/paper/2408.13454/) (2024)
- [SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset](https://mlsanity.com/paper/2502.01894/) (2025)
- [S2GO: Streaming Sparse Gaussian Occupancy Prediction](https://mlsanity.com/paper/2506.05473/) (2025)
- [MambaMap: Online Vectorized HD Map Construction using State Space Model](https://mlsanity.com/paper/2507.20224/) (2025)
- [LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation](https://mlsanity.com/paper/2512.20217/) (2025)
- [OnlineBEV: Recurrent Temporal Fusion in Bird's Eye View Representations for Multi-Camera 3D Perception](https://mlsanity.com/paper/2507.08644/) (2025)
- [SimPB++: Simultaneously Detecting 2D and 3D Objects from Multiple Cameras](https://mlsanity.com/paper/2605.01924/) (2026)

## Abstract (arXiv)

3D object detection with surround-view images is an essential task for autonomous driving. In this work, we propose DETR4D, a Transformer-based framework that explores sparse attention and direct feature query for 3D object detection in multi-view images. We design a novel projective cross-attention mechanism for query-image interaction to address the limitations of existing methods in terms of geometric cue exploitation and information loss for cross-view objects. In addition, we introduce a heatmap generation technique that bridges 3D and 2D spaces efficiently via query initialization. Furthermore, unlike the common practice of fusing intermediate spatial features for temporal aggregation, we provide a new perspective by introducing a novel hybrid approach that performs cross-frame fusion over past object queries and image features, enabling efficient and robust modeling of temporal information. Extensive experiments on the nuScenes dataset demonstrate the effectiveness and efficiency of the proposed DETR4D.