mlsanityData to 14 Oct 2026

Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

arXiv 2604.16395 ↗First posted 29 Mar 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Context retrieval systems for LLM inference face a critical challenge: high retrieval latency creates a fundamental tension between waiting for complete context (poor time-to-first-token) and proceeding without it (reduced quality). Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency, but doing so with concurrent requests introduces new challenges: requests contend for GPU compute and memory, and scheduling must adapt to dynamic context arrivals. We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments. Stream2LLM introduces adaptive scheduling and preemption for two distinct retrieval patterns: append-mode (progressive context accumulation) and update-mode (iterative refinement with cache invalidation). It decouples scheduling decisions from resource acquisition, enabling flexible preemption strategies guided by hardware-specific cost models, and uses longest common prefix matching to minimize redundant computation when input changes dynamically. To evaluate Stream2LLM, we collect two large-scale, real-world streaming workloads based on web crawling and approximate nearest neighbor search. Our evaluation demonstrates that streaming architecture delivers up to 11x TTFT improvements, with cost-aware scheduling providing critical benefits under memory pressure, all while maintaining throughput parity with non-streaming baselines. Code: https://github.com/rajveerb/stream2llm/tree/mlsys_artifact

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 23 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

15 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 11

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024Measuring short-form factuality in large language modelscited by 373
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  10. 2023Retrieval meets Long Context Large Language Modelscited by 115
  11. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
  2. 2025SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs
  3. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
  4. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  5. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving