mlsanityData to 7 Oct 2026

PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection

arXiv 2603.21576 ↗First posted 23 Mar 2026physics.optics cs.AI cs.ARcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLM inference is bottlenecked not by compute but by the O(n) memory bandwidth cost of scanning the KV cache at every decode step -- a wall that no amount of arithmetic scaling can break. Recent photonic accelerators have demonstrated impressive throughput for dense attention computation; however, these approaches inherit the same O(n) memory scaling as electronic attention when applied to long contexts. We observe that the real leverage point is the coarse block-selection step: a memory-bound similarity search that determines which KV blocks to fetch. We identify, for the first time, that this task is structurally matched to the photonic broadcast-and-weight paradigm -- the query fans out to all candidates via passive splitting, signatures are quasi-static (matching electro-optic MRR programming), and only rank order matters (relaxing precision to 4-6 bits). Crucially, the photonic advantage grows with context length: as N increases, the electronic scan cost rises linearly while the photonic evaluation remains O(1). We instantiate this insight in PRISM (Photonic Ranking via Inner-product Similarity with Microring weights), a thin-film lithium niobate (TFLN) similarity engine. Hardware-impaired needle-in-a-haystack evaluation on Qwen2.5-7B confirms 100% accuracy from 4K through 64K tokens at k=32, with 16x traffic reduction at 64K context. PRISM achieves a four-order-of-magnitude energy advantage over GPU baselines at practical context lengths (n >= 4K).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 86 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

26 of 86 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference19 May
  2. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  5. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  11. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  12. 2020Photonics for artificial intelligence and neuromorphic computingcited by 111
6 more
  1. 2024AI and Memory Wallcited by 89
  2. 2016Neuromorphic Silicon Photonic Networkscited by 66
  3. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  4. 2021CrossLight: A Cross-Layer Optimized Silicon Photonic Neural Network Acceleratorcited by 29
  5. 2023Lightening-Transformer: A Dynamically-operated Optically-interconnected Photonic Transformer Acceleratorcited by 15
  6. 2023High-Speed and Energy-Efficient Non-Volatile Silicon Photonic Memory Based on Heterogeneously Integrated Memresonatorcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
  2. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  3. 2026AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
  4. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
  5. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  6. 2026ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
  7. 2026Selective KV Cache Protection for Noise-Resilient LLM Inference on Analog Compute-In-Memory Systems
  8. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference