mlsanityData to 14 Oct 2026

SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

arXiv 2605.14110 ↗First posted 13 May 2026cs.CV cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D vision, prune or merge image tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other embeddings. Evaluated on nuScenes and our new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical agents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 60 of its 63 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

60 of 63 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
38 more
  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  2. 2022Planning-oriented Autonomous Drivingcited by 1,087
  3. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  4. 2022Token Merging: Your ViT But Fastercited by 1,021
  5. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  10. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 658
  11. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  12. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  13. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  14. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 533
  15. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
  16. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  17. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  18. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  19. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  20. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 241
  21. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  22. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  23. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 178
  24. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  25. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  26. 2019An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detectioncited by 163
  27. 2023SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videoscited by 150
  28. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  29. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  30. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  31. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  32. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  33. 2023Sparse4D v2: Recurrent Temporal Fusion with Sparse Modelcited by 93
  34. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 68
  35. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  36. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  37. 2020Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inferencecited by 34
  38. 2024Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinementcited by 31

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.