# Fully Sparse Fusion for 3D Object Detection

arXiv: https://arxiv.org/abs/2304.12310 · HTML: https://mlsanity.com/paper/2304.12310/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 82 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 30 of 69)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)

## Built on by (most cited first, 29 of 29)

- [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024)
- [End-to-End Driving with Online Trajectory Evaluation via BEV World Model](https://mlsanity.com/paper/2504.01941/) (2025)
- [UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation](https://mlsanity.com/paper/2308.07732/) (2023)
- [MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detection](https://mlsanity.com/paper/2408.05945/) (2024)
- [DeepInteraction++: Multi-Modality Interaction for Autonomous Driving](https://mlsanity.com/paper/2408.05075/) (2024)
- [Fusion is Not Enough: Single Modal Attacks on Fusion Models for 3D Object Detection](https://mlsanity.com/paper/2304.14614/) (2023)
- [Progressive Multi-Modal Fusion for Robust 3D Object Detection](https://mlsanity.com/paper/2410.07475/) (2024)
- [Radars for Autonomous Driving: A Review of Deep Learning Methods and Challenges](https://mlsanity.com/paper/2306.09304/) (2023)
- [FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving](https://mlsanity.com/paper/2408.06832/) (2024)
- [SparseFusion: Efficient Sparse Multi-Modal Fusion Framework for Long-Range 3D Perception](https://mlsanity.com/paper/2403.10036/) (2024)
- [How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation](https://mlsanity.com/paper/2505.18956/) (2025)
- [Self-Supervised Sparse Sensor Fusion for Long Range Perception](https://mlsanity.com/paper/2508.13995/) (2025)
- [Towards Long-Range 3D Object Detection for Autonomous Vehicles](https://mlsanity.com/paper/2310.04800/) (2023)
- [PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interest](https://mlsanity.com/paper/2403.09212/) (2024)
- [MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation](https://mlsanity.com/paper/2410.08739/) (2024)
- [CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2603.23276/) (2026)
- [DSERT-RoLL: Robust Multi-Modal Perception for Diverse Driving Conditions with Stereo Event-RGB-Thermal Cameras, 4D Radar, and Dual-LiDAR](https://mlsanity.com/paper/2604.03685/) (2026)
- [What You See Is What You Detect: Towards better Object Densification in 3D detection](https://mlsanity.com/paper/2310.17842/) (2023)
- [FSMDet: Vision-guided feature diffusion for fully sparse 3D detector](https://mlsanity.com/paper/2409.06945/) (2024)
- [MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2602.08126/) (2026)
- [Reliable Multimodal Learning Via Multi-Level Adaptive DeConfusion](https://mlsanity.com/paper/2502.19674/) (2025)
- [Rethink 3D Object Detection from Physical World](https://mlsanity.com/paper/2507.00190/) (2025)
- [Co-Win: Joint Object Detection and Instance Segmentation in LiDAR Point Clouds via Collaborative Window Processing](https://mlsanity.com/paper/2507.19691/) (2025)
- [CrossRay3D: Geometry and Distribution Guidance for Efficient Multimodal 3D Detection](https://mlsanity.com/paper/2510.15991/) (2025)
- [Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification](https://mlsanity.com/paper/2601.07163/) (2026)
- [Information Coordination as a Bridge: A Neuro-Symbolic Architecture for Reliable Autonomous Driving Scene Understanding](https://mlsanity.com/paper/2605.04475/) (2026)
- [GaussianFusion: Unified 3D Gaussian Representation for Multi-Modal Fusion Perception](https://mlsanity.com/paper/2607.00746/) (2026)
- [DeGuNet: Depth-Guided Ultra-Compact Backbones for Efficient LiDAR-Camera 3D Detection](https://mlsanity.com/paper/2607.12419/) (2026)
- [RSFusionDet: Underwater RGB-Sonar Multimodal Object Detection](https://mlsanity.com/paper/2608.25367/) (2026)

## Abstract (arXiv)

Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View (BEV) feature maps. However, the cost of such BEV feature maps is quadratic to the detection range, making it not suitable for long-range detection. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture. Particularly, utilizing instance queries, our framework integrates the well-studied 2D instance segmentation into the LiDAR side, which is parallel to the 3D instance segmentation part in the fully sparse detector. This design achieves a uniform query-based fusion framework in both the 2D and 3D sides while maintaining the fully sparse characteristic. Extensive experiments showcase state-of-the-art results on the widely used nuScenes dataset and the long-range Argoverse 2 dataset. Notably, the inference speed of the proposed method under the long-range LiDAR perception setting is 2.7 × faster than that of other state-of-the-art multimodal 3D detection methods. Code will be released at urlhttps://github.com/BraveGroup/FullySparseFusion.