mlsanityData to 14 Oct 2026

Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction

arXiv 2504.12959 ↗First posted 17 Apr 2025cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

We present GDFusion, a temporal fusion method for vision-based 3D semantic occupancy prediction (VisionOcc). GDFusion opens up the underexplored aspects of temporal fusion within the VisionOcc framework, focusing on both temporal cues and fusion strategies. It systematically examines the entire VisionOcc pipeline, identifying three fundamental yet previously overlooked temporal cues: scene-level consistency, motion calibration, and geometric complementation. These cues capture diverse facets of temporal evolution and make distinct contributions across various modules in the VisionOcc framework. To effectively fuse temporal signals across heterogeneous representations, we propose a novel fusion strategy by reinterpreting the formulation of vanilla RNNs. This reinterpretation leverages gradient descent on features to unify the integration of diverse temporal information, seamlessly embedding the proposed temporal cues into the network. Extensive experiments on nuScenes demonstrate that GDFusion significantly outperforms established baselines. Notably, on Occ3D benchmark, it achieves 1.4%-4.8% mIoU improvements and reduces memory consumption by 27%-72%.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 66 of its 75 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

66 of 75 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  11. 2017Deformable Convolutional Networkscited by 1,488
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
38 more
  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  4. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  5. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  6. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  7. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  9. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  10. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  11. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 352
  12. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 341
  13. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  14. 2024Titans: Learning to Memorize at Test Timecited by 328
  15. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  16. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  17. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  18. 2018How to train your MAMLcited by 309
  19. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  20. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  21. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  22. 2023Scene as Occupancycited by 209
  23. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 178
  24. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 165
  25. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 157
  26. 2023StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Constructioncited by 145
  27. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 128
  28. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 127
  29. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 126
  30. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
  31. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 103
  32. 2022The Devil in Linear Transformercited by 98
  33. 2023COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Predictioncited by 96
  34. 2023Sparse4D v2: Recurrent Temporal Fusion with Sparse Modelcited by 93
  35. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
  36. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 78
  37. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 67
  38. 2023Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perceptioncited by 65

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Semantic Causality-Aware Vision-Based 3D Occupancy Predictioncited by 8
  2. 2025OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspectivecited by 5
  3. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  4. 2026SuperOcc: Toward Cohesive Temporal Modeling for Superquadric-based 3D Occupancy Predictioncited by 1
  5. 2026PolarVSR: A Unified Framework and Benchmark for Continuous Space-Time Polarization Video Reconstructioncited by 1
  6. 2026SGR-OCC: Evolving Monocular Priors for Embodied 3D Occupancy Prediction via Soft-Gating Lifting and Semantic-Adaptive Geometric Refinementcited by 0
  7. 2026Height-Guided Projection Reparameterization for Camera-LiDAR Occupancycited by 0
  8. 2026FDR-Occ: Factorized Dense Routing for Full-Spectrum 3D Occupancy Predictioncited by 0
  9. 2026RoadOcc Learns When to Persist, Transport, or Refresh Memory for Roadside Occupancy Predictioncited by 0