mlsanityData to 7 Oct 2026

Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

arXiv 2502.02789 ↗First posted 5 Feb 2025cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7× maximal end-to-end QPS on real downstream tasks and 7.66× TTFT improvement.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

34 of 52 references.

References · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
22 more
  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  13. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  14. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  15. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  16. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  17. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  18. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  19. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reductioncited by 39
  20. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  21. 2024KV Prediction for Improved Time to First Tokencited by 9
  22. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 10
  2. 2025REFRAG: Rethinking RAG based Decodingcited by 8
  3. 2025Draft-based Approximate Inference for LLMscited by 6
  4. 2025HAMburger: Accelerating LLM Inference via Token Smashingcited by 4
  5. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  6. 2026When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judgescited by 2
  7. 2026Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Modelscited by 2
  8. 2025Don't Pay Attentioncited by 1
  9. 2026CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefillcited by 0
  10. 2026LongAttnComp: Cross-Family Context Compression for Long-Context Reasoningcited by 0
  11. 2026Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Modelscited by 0
  12. 2026Audio Token Attention Is Predictable Before the Language Model Runscited by 0