# Sparse4D v2: Recurrent Temporal Fusion with Sparse Model

arXiv: https://arxiv.org/abs/2305.14018 · HTML: https://mlsanity.com/paper/2305.14018/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 34 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision](https://mlsanity.com/paper/2211.10439/) (2022)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection](https://mlsanity.com/paper/2210.02443/) (2022)
- [An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection](https://mlsanity.com/paper/1904.09730/) (2019)
- [Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion](https://mlsanity.com/paper/2211.10581/) (2022)
- [Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud](https://mlsanity.com/paper/1903.09847/) (2019)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo](https://mlsanity.com/paper/2209.10248/) (2022)
- [BEVPoolv2: A Cutting-edge Implementation of BEVDet Toward Deployment](https://mlsanity.com/paper/2211.17111/) (2022)
- [BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2211.09386/) (2022)

## Built on by (most cited first, 30 of 93)

- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos](https://mlsanity.com/paper/2308.09244/) (2023)
- [StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Construction](https://mlsanity.com/paper/2308.12570/) (2023)
- [FB-BEV: BEV Representation from Forward-Backward View Transformations](https://mlsanity.com/paper/2308.02236/) (2023)
- [Far3D: Expanding the Horizon for Surround-view 3D Object Detection](https://mlsanity.com/paper/2308.09616/) (2023)
- [Sparse4D v3: Advancing End-to-End 3D Detection and Tracking](https://mlsanity.com/paper/2311.11722/) (2023)
- [MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping](https://mlsanity.com/paper/2403.15951/) (2024)
- [OPUS: Occupancy Prediction Using a Sparse Set](https://mlsanity.com/paper/2409.09350/) (2024)
- [SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving](https://mlsanity.com/paper/2404.06892/) (2024)
- [Stream Query Denoising for Vectorized HD Map Construction](https://mlsanity.com/paper/2401.09112/) (2024)
- [Ray Denoising: Depth-aware Hard Negative Sampling for Multi-view 3D Object Detection](https://mlsanity.com/paper/2402.03634/) (2024)
- [BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection](https://mlsanity.com/paper/2312.01696/) (2023)
- [HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder](https://mlsanity.com/paper/2503.08612/) (2025)
- [ADA-Track++: End-to-End Multi-Camera 3D Multi-Object Tracking with Alternating Detection and Association](https://mlsanity.com/paper/2405.08909/) (2024)
- [SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2403.07284/) (2024)
- [MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detection](https://mlsanity.com/paper/2408.05945/) (2024)
- [PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors](https://mlsanity.com/paper/2403.09079/) (2024)
- [Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?](https://mlsanity.com/paper/2405.18361/) (2024)
- [Neural Radiance Field in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.13816/) (2024)
- [HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras](https://mlsanity.com/paper/2404.02517/) (2024)
- [UAV3D: A Large-scale 3D Perception Benchmark for Unmanned Aerial Vehicles](https://mlsanity.com/paper/2410.11125/) (2024)
- [CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow](https://mlsanity.com/paper/2403.08919/) (2024)
- [CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception](https://mlsanity.com/paper/2503.13504/) (2025)
- [DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale](https://mlsanity.com/paper/2604.00813/) (2026)
- [OPEN: Object-wise Position Embedding for Multi-view 3D Object Detection](https://mlsanity.com/paper/2407.10753/) (2024)
- [PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2407.17378/) (2024)
- [GeoBEV: Learning Geometric BEV Representation for Multi-view 3D Object Detection](https://mlsanity.com/paper/2409.01816/) (2024)
- [SimPB: A Single Model for 2D and 3D Object Detection from Multiple Cameras](https://mlsanity.com/paper/2403.10353/) (2024)
- [RayFormer: Improving Query-Based Multi-Camera 3D Object Detection via Ray-Centric Strategies](https://mlsanity.com/paper/2407.14923/) (2024)

## Abstract (arXiv)

Sparse algorithms offer great flexibility for multi-view temporal perception tasks. In this paper, we present an enhanced version of Sparse4D, in which we improve the temporal fusion module by implementing a recursive form of multi-frame feature sampling. By effectively decoupling image features and structured anchor features, Sparse4D enables a highly efficient transformation of temporal features, thereby facilitating temporal fusion solely through the frame-by-frame transmission of sparse features. The recurrent temporal fusion approach provides two main benefits. Firstly, it reduces the computational complexity of temporal fusion from O(T) to O(1), resulting in significant improvements in inference speed and memory usage. Secondly, it enables the fusion of long-term information, leading to more pronounced performance improvements due to temporal fusion. Our proposed approach, Sparse4Dv2, further enhances the performance of the sparse perception algorithm and achieves state-of-the-art results on the nuScenes 3D detection benchmark. Code will be available at urlhttps://github.com/linxuewu/Sparse4D.