mlsanityData to 7 Oct 2026

GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

arXiv 2605.15852 ↗First posted 15 May 2026cs.CVcited by 2

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Streaming 3D reconstruction from long monocular video sequences requires maintaining a key-value (KV) cache that grows linearly with sequence length, creating a severe memory bottleneck. Existing approaches either truncate the cache to a fixed set of anchor frames, leading to reconstruction quality degradation, or rely on attention-score heuristics that are agnostic to 3D scene structure, failing to preserve geometrically valuable tokens. To address these problems, we present GHOST (Geometry-Hierarchical Online Streaming Token Eviction), a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant tokens online. GHOST introduces three mutually reinforcing innovations: a hierarchical dual-level importance scoring scheme, a privilege mechanism that protects special tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation. Experiments on various benchmarks show that GHOST preserves excellent reconstruction quality while cutting the KV cache by nearly half and delivering 1.75x faster inference compared to state-of-the-art methods. Our code is available at https://github.com/lokiniuniu/GHOST.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 37 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

35 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers22 Sep
  2. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  3. 2022Token Merging: Your ViT But Faster17 Oct
  4. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  6. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  11. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
19 more
  1. 2021Neural RGB-D Surface Reconstructioncited by 312
  2. 2019ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residualscited by 180
  3. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  4. 2025Streaming 4D Visual Geometry Transformercited by 147
  5. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  6. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 89
  7. 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 82
  8. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  9. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 58
  10. 2026InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streamscited by 51
  11. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  12. 2025QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generationcited by 20
  13. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMscited by 6
  14. 2022EViT: Privacy-Preserving Image Retrieval via Encrypted Vision Transformer in Cloud Computingcited by 5
  15. 2025Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspectivecited by 5
  16. 2025FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editingcited by 4
  17. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  18. 2025BalanceGS: Algorithm-System Co-design for Efficient 3D Gaussian Splatting Training on GPUcited by 2
  19. 2024Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Priorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstructioncited by 0
  2. 2026Slot3R: Set-Associative Spatial Memory for Streaming 3D Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer
  2. 2026FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry
  3. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
  4. 2026XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
  5. 2026Stream3D: Sequential Multi-View 3D Generation via Evidential Memory
  6. 2026LongStream: Long-Sequence Streaming Autoregressive Visual Geometry
  7. 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
  8. 2026Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction