mlsanityData to 13 Oct 2026

Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy

arXiv 2312.12728 ↗First posted 20 Dec 2023cs.IR cs.AI cs.LGcited by 37

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As Large Language Models (LLMs) have made significant advancements across various tasks, such as question answering, translation, text summarization, and dialogue systems, the need for accuracy in information becomes crucial, especially for serious financial products serving billions of users like Alipay. However, for a real-world product serving millions of users, the inference speed of LLMs becomes a critical factor compared to a mere experimental model. Hence, this paper presents a generic framework for accelerating the inference process, resulting in a substantial increase in speed and cost reduction for our LLM-based scenarios, with lossless generation accuracy. In the traditional inference process, each token is generated sequentially by the LLM, leading to a time consumption proportional to the number of generated tokens. To enhance this process, our framework, named textitlookahead, introduces a textitmulti-branch strategy. Instead of generating a single token at a time, we propose a Trie-based retrieval and verification mechanism to be able to accept several tokens at a forward step. Our strategy offers two distinct advantages: (1) it guarantees absolute correctness of the output, avoiding any approximation algorithms, and (2) the worst-case performance of our approach is equivalent to the conventional process. We conduct extensive experiments to demonstrate the significant improvements achieved by applying our inference acceleration framework. Our framework is widely deployed in Alipay since April 2023, and obtain remarkable 2.66x to 6.26x speedup. Our code is available at https://github.com/alipay/PainlessInferenceAcceleration.

Key facts. Cited by 37 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 48 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

28 of 48 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 990
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 719
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 668
16 more
  1. 2017Non-Autoregressive Neural Machine Translationcited by 462
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 361
  3. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 326
  4. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 319
  5. 2021Block Pruning For Faster Transformerscited by 176
  6. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  7. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  8. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  9. 2019A Tensorized Transformer for Language Modelingcited by 86
  10. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  11. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  12. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  13. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  14. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
  15. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 26
  16. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 16.

37 papers.

Sort:

By area:

  1. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 59
  2. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  3. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  4. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  5. 2025Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningcited by 10
  6. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 10
  7. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  8. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  9. 2024When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMscited by 6
  10. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6
  11. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 6
  12. 2026Speculative Decoding with a Speculative Vocabularycited by 6
25 more
  1. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  2. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  3. 2025Scaling Laws for Speculative Decodingcited by 4
  4. 2025HAMburger: Accelerating LLM Inference via Token Smashingcited by 4
  5. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  6. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 4
  7. 2026PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decodingcited by 4
  8. 2025Cacheback: Speculative Decoding With Nothing But Cachecited by 3
  9. 2026FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decodingcited by 3
  10. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Modelscited by 3
  11. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  12. 2025Zero-Shot Cross-Domain Code Search without Fine-Tuningcited by 2
  13. 2026JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergencecited by 2
  14. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  15. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernelscited by 1
  16. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
  17. 2025Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Treescited by 1
  18. 2026AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decodingcited by 1
  19. 2026To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signalscited by 1
  20. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inferencecited by 0
  21. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inferencecited by 0
  22. 2026Quantifying Hallucinations in Language Language Models on Medical Textbookscited by 0
  23. 2026FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decodingcited by 0
  24. 2026ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decodingcited by 0
  25. 2026TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  2. 2024A Survey on Efficient Inference for Large Language Models
  3. 2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Models
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  5. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  6. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  7. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights