# FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

arXiv: https://arxiv.org/abs/2606.09079 · HTML: https://mlsanity.com/paper/2606.09079/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 3 of its 5 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 3 of 3)

- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)

## Built on by (most cited first, 4 of 4)

- [Semantic-aware and Self-improving Program Reduction via Agentic Large Language Models](https://mlsanity.com/paper/2607.03766/) (2026)
- [Variational-Ising-Attention:Tailored Attention Matters for Science](https://mlsanity.com/paper/2607.23634/) (2026)
- [OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching](https://mlsanity.com/paper/2608.08097/) (2026)
- [KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression](https://mlsanity.com/paper/2610.08811/) (2026)

## Abstract (arXiv)

Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose Lookahead Sparse Attention (LSA), a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a backbone-free decoupled training strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this ``less is more'' paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), textttFM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6% absolute margin on average). At 1M context, per-decode-token compute drops to 0.30× of the baseline and GPU KV cache shrinks by 90% (3.73→0.37 GB), translating into 2.8× aggregate throughput and 2.7× concurrency gains in PD-disaggregated serving on 8×H20 GPUs.