mlsanityData to 7 Oct 2026

PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

arXiv 2505.07203 ↗First posted 12 May 2025cs.DCcited by 21

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks, such as recommendation, credit verification, and data labeling. The key characteristic of these emerging use cases is that the LLM generates only a single output token, rather than an arbitrarily long sequence of tokens. We call this prefill-only workload. However, since existing LLM engines assume arbitrary output lengths, they fail to leverage the unique properties of prefill-only workloads. In this paper, we present PrefillOnly, the first LLM inference engine that improves the inference throughput and latency by fully embracing the properties of prefill-only workloads. First, since it generates only one token, PrefillOnly only needs to store the KV cache of only the last computed layer, rather than of all layers. This drastically reduces the GPU memory footprint of LLM inference and allows handling long inputs without using solutions that reduces throughput, such as cross-GPU KV cache parallelization. Second, because the output length is fixed, rather than arbitrary, PrefillOnly can precisely determine the job completion time (JCT) of each prefill-only request before it starts. This enables efficient JCT-aware scheduling policies such as shortest remaining job first. PrefillOnly can process upto 4x larger queries per second without inflating average and P99 latency.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 71 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

39 of 71 references.

References · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2023A Survey on Large Language Models for Recommendationcited by 378
  12. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
27 more
  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  3. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  5. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  6. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  7. 2025OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignmentcited by 167
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  9. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  10. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  11. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  12. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  13. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  14. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
  15. 2023Enhancing Recommender Systems with Large Language Model Reasoning Graphscited by 52
  16. 2023RAH! RecSys-Assistant-Human: A Human-Centered Recommendation Framework with LLM Agentscited by 30
  17. 2024LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Applicationcited by 21
  18. 2023Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Modelscited by 20
  19. 2024Depression Detection on Social Media with Large Language Modelscited by 20
  20. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Modelscited by 18
  21. 2024Do Large Language Models Need a Content Delivery Network?cited by 15
  22. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15
  23. 2023Spam-T5: Benchmarking Large Language Models for Few-Shot Email Spam Detectioncited by 12
  24. 2025360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendationcited by 12
  25. 2025Towards Large-scale Generative Rankingcited by 10
  26. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
  27. 2023OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 11.

21 papers.

Sort:

By area:

  1. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  2. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  3. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  4. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  5. 2025TetriServe: Efficiently Serving Mixed DiT Workloadscited by 4
  6. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  7. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 3
  8. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  9. 2025xGR: Efficient Generative Recommendation Serving at Scalecited by 2
  10. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  11. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
  12. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
9 more
  1. 2025On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwardingcited by 1
  2. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  3. 2025HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inferencecited by 0
  4. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  5. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demandscited by 0
  6. 2026Designing Datacenter Power Delivery Hierarchies for the AI Eracited by 0
  7. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  8. 2026MOCAP: Wafer-Scale-Chip-Oriented Memory-Orchestrated Chunked Pipelining Framework for Prefill-Only LLM Inferencecited by 0
  9. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  2. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  3. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation
  4. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  5. 2024An Efficient Inference Framework for Early-exit Large Language Models
  6. 2026FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
  7. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  8. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading