mlsanityData to 14 Oct 2026

Inference with Reference: Lossless Acceleration of Large Language Models

arXiv 2304.04487 ↗First posted 10 Apr 2023cs.CL cs.AIcited by 65

Not on the map. Why?

Cited by 43 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). LLMA first selects a text span from the reference and copies its tokens to the decoder and then efficiently checks the tokens' appropriateness as the decoding result in parallel within one decoding step. The improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in many practical generation scenarios where significant overlap between in-context reference and outputs exists (e.g., search engines and multi-turn conversations).

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

11 of 14 references.

References · 11

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  9. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  10. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  11. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 30; 2025: 10; 2026: 14.

65 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  5. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  6. 2023Efficient Large Language Models: A Surveycited by 186
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  9. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  10. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  12. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 75
38 more
  1. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  2. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  3. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  4. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  5. 2024A Survey on Transformer Compressioncited by 46
  6. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  7. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  8. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 37
  9. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 36
  10. 2024Block Verification Accelerates Speculative Decodingcited by 30
  11. 2023Copy Is All You Needcited by 27
  12. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 25
  13. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  14. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  15. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  16. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  17. 2023Large Search Model: Redefining Search Stack in the Era of LLMscited by 18
  18. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  19. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  20. 2023Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decodingcited by 10
  21. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledgecited by 10
  22. 2025FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edgecited by 10
  23. 2024Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 8
  24. 2023On the Opportunities of Green Computing: A Surveycited by 7
  25. 2024CursorCore: Assist Programming through Aligning Anythingcited by 7
  26. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  27. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  28. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  29. 2024Clover-2: Accurate Inference for Regressive Lightweight Speculative Decodingcited by 5
  30. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 5
  31. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 5
  32. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  33. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  34. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  35. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  36. 2023SCALE: Synergized Collaboration of Asymmetric Language Translation Enginescited by 3
  37. 2024Hardware Acceleration of LLMs: A comprehensive survey and comparisoncited by 3
  38. 2026FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decodingcited by 3

50 most cited of 65 All 65 (JSON)