mlsanityData to 7 Oct 2026

FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving

arXiv 2408.06832 ↗First posted 13 Aug 2024cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

The integration of data from diverse sensor modalities (e.g., camera and LiDAR) constitutes a prevalent methodology within the ambit of autonomous driving scenarios. Recent advancements in efficient point cloud transformers have underscored the efficacy of integrating information in sparse formats. When it comes to fusion, since image patches are dense in pixel space with ambiguous depth, it necessitates additional design considerations for effective fusion. In this paper, we conduct a comprehensive exploration of design choices for Transformer-based sparse cameraLiDAR fusion. This investigation encompasses strategies for image-to-3D and LiDAR-to-2D mapping, attention neighbor grouping, single modal tokenizer, and micro-structure of Transformer. By amalgamating the most effective principles uncovered through our investigation, we introduce FlatFusion, a carefully designed framework for sparse camera-LiDAR fusion. Notably, FlatFusion significantly outperforms state-of-the-art sparse Transformer-based methods, including UniTR, CMT, and SparseFusion, achieving 73.7 NDS on the nuScenes validation set with 10.1 FPS with PyTorch.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 70 of its 93 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

70 of 93 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
38 more
  1. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  2. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  4. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  5. 2022Planning-oriented Autonomous Drivingcited by 1,068
  6. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  7. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  8. 2020PCT: Point cloud transformercited by 730
  9. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  10. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  11. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  12. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  13. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  14. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  15. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  16. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  17. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  18. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  19. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  20. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  21. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  22. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  23. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  24. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  25. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  26. 2021Multimodal Virtual Point 3D Detectioncited by 166
  27. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  28. 2019An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detectioncited by 163
  29. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  30. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  31. 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
  32. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  33. 2023DSVT: Dynamic Sparse Voxel Transformer with Rotated Setscited by 122
  34. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
  35. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  36. 2023FB-BEV: BEV Representation from Forward-Backward View Transformationscited by 98
  37. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  38. 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 1.

9 papers.

  1. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  2. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  3. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  4. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  5. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5
  6. 2025DriveVGGT: Calibration-Constrained Visual Geometry Transformers for Multi-Camera Autonomous Drivingcited by 4
  7. 2025MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detectioncited by 3
  8. 2025A Novel Vision Transformer for Camera-LiDAR Fusion based Traffic Object Segmentationcited by 1
  9. 2026Variational Inference for Bird's Eye View Segmentation in Autonomous Drivingcited by 0