mlsanityData to 14 Oct 2026

OverFill: Two-Stage Models for Efficient Language Model Decoding

arXiv 2508.08446 ↗First posted 11 Aug 2025cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

46 of 54 references.

References · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
34 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  12. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  13. 2022Confident Adaptive Language Modelingcited by 310
  14. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  15. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  16. 2024Are We Done with MMLU?cited by 194
  17. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  18. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  19. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  20. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  21. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  22. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  23. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  24. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  25. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  26. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  27. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 75
  28. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  29. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  30. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  31. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
  32. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decodingcited by 11
  33. 2024Tandem Transformers for Inference Efficient LLMscited by 10
  34. 2024Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signalscited by 6
  2. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0