# 3D-MAN: 3D Multi-frame Attention Network for Object Detection

arXiv: https://arxiv.org/abs/2103.16054 · HTML: https://mlsanity.com/paper/2103.16054/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Multi-Task Multi-Sensor Fusion for 3D Object Detection](https://mlsanity.com/paper/2012.12397/) (2020)
- [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Pillar-based Object Detection for Autonomous Driving](https://mlsanity.com/paper/2007.10323/) (2020)
- [Memory Enhanced Global-Local Aggregation for Video Object Detection](https://mlsanity.com/paper/2003.12063/) (2020)
- [Sequence Level Semantics Aggregation for Video Object Detection](https://mlsanity.com/paper/1907.06390/) (2019)
- [Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection](https://mlsanity.com/paper/2005.09927/) (2020)
- [Relation Distillation Networks for Video Object Detection](https://mlsanity.com/paper/1908.09511/) (2019)
- [LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention](https://mlsanity.com/paper/2004.01389/) (2020)
- [AFDet: Anchor Free One Stage 3D Object Detection](https://mlsanity.com/paper/2006.12671/) (2020)
- [IPOD: Intensive Point-based Object Detector for Point Cloud](https://mlsanity.com/paper/1812.05276/) (2018)
- [StarNet: Targeted Computation for Object Detection in Point Clouds](https://mlsanity.com/paper/1908.11069/) (2019)

## Built on by (most cited first, 30 of 73)

- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.11301/) (2023)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection](https://mlsanity.com/paper/2205.07403/) (2022)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [Visual Point Cloud Forecasting enables Scalable Autonomous Driving](https://mlsanity.com/paper/2312.17655/) (2023)
- [AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds](https://mlsanity.com/paper/2112.09205/) (2021)
- [Fully Sparse 3D Object Detection](https://mlsanity.com/paper/2207.10035/) (2022)
- [LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion](https://mlsanity.com/paper/2303.03595/) (2023)
- [Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection](https://mlsanity.com/paper/2406.10700/) (2024)
- [MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection](https://mlsanity.com/paper/2205.05979/) (2022)
- [Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perception](https://mlsanity.com/paper/2303.05970/) (2023)
- [Fully Sparse 3D Occupancy Prediction](https://mlsanity.com/paper/2312.17118/) (2023)
- [GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds](https://mlsanity.com/paper/2212.03010/) (2022)
- [Dynamic 3D Scene Analysis by Point Cloud Accumulation](https://mlsanity.com/paper/2207.12394/) (2022)
- [FastPillars: A Deployment-friendly Pillar-based 3D Detector](https://mlsanity.com/paper/2302.02367/) (2023)
- [A Unified Query-based Paradigm for Point Cloud Understanding](https://mlsanity.com/paper/2203.01252/) (2022)
- [Super Sparse 3D Object Detection](https://mlsanity.com/paper/2301.02562/) (2023)
- [BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregation](https://mlsanity.com/paper/2303.17099/) (2023)
- [DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds](https://mlsanity.com/paper/2306.06023/) (2023)
- [Fully Sparse Fusion for 3D Object Detection](https://mlsanity.com/paper/2304.12310/) (2023)
- [Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection](https://mlsanity.com/paper/2304.12315/) (2023)
- [Densely Constrained Depth Estimator for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10047/) (2022)
- [From Multi-View to Hollow-3D: Hallucinated Hollow-3D R-CNN for 3D Object Detection](https://mlsanity.com/paper/2107.14391/) (2021)
- [PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer](https://mlsanity.com/paper/2305.06621/) (2023)
- [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024)
- [CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection](https://mlsanity.com/paper/2411.03013/) (2024)

## Abstract (arXiv)

3D object detection is an important module in autonomous driving and robotics. However, many existing methods focus on using single frames to perform 3D detection, and do not fully utilize information from multiple frames. In this paper, we present 3D-MAN: a 3D multi-frame attention network that effectively aggregates features from multiple perspectives and achieves state-of-the-art performance on Waymo Open Dataset. 3D-MAN first uses a novel fast single-frame detector to produce box proposals. The box proposals and their corresponding feature maps are then stored in a memory bank. We design a multi-view alignment and aggregation module, using attention networks, to extract and aggregate the temporal features stored in the memory bank. This effectively combines the features coming from different perspectives of the scene. We demonstrate the effectiveness of our approach on the large-scale complex Waymo Open Dataset, achieving state-of-the-art results compared to published single-frame and multi-frame methods.