mlsanityData to 13 Oct 2026

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

arXiv 2408.08696 ↗First posted 16 Aug 2024cs.CL cs.LGcited by 47

Not on the map. Why?

Cited by 39 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Massive parameters of LLMs have made inference latency a fundamental bottleneck. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm. Some methods rely on additional architectures to guess draft tokens, which need extra training before use. Alternatively, retrieval-based training-free techniques build libraries from pre-existing corpora or by n-gram generation. However, they face challenges like large storage requirements, time-consuming retrieval, and limited adaptability. Observing that candidate tokens generated during the decoding process are likely to reoccur in future sequences, we propose Token Recycling. It stores candidate tokens in an adjacency matrix and employs a breadth-first-search (BFS)-like algorithm to construct a draft tree, which is then validated through tree attention. New candidate tokens from the decoding process are then used to update the matrix. Token Recycling requires textless2MB of additional storage and achieves approximately 2x speedup across all sizes of LLMs. It significantly outperforms existing train-free methods by 30% and even a widely recognized training method by 25%.

Key facts. Cited by 47 papers in the map (data through 13 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

24 of 36 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  5. 2021Program Synthesis with Large Language Modelscited by 3,573
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,218
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
12 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 990
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 569
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 361
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  8. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  9. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  10. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  11. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  12. 2024Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Trainingcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 26.

47 papers.

Sort:

By area:

  1. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 59
  2. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  3. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  4. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  5. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 21
  6. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  7. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 14
  8. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 13
  9. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 10
  10. 2025CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Qualitycited by 6
  11. 2025DReSD: Dense Retrieval for Speculative Decodingcited by 6
  12. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
35 more
  1. 2025Parallel Token Prediction for Language Modelscited by 6
  2. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  3. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  4. 2025Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decodingcited by 4
  5. 2025SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verificationcited by 4
  6. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMscited by 4
  7. 2026ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decodingcited by 4
  8. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decodingcited by 4
  9. 2025When, What, and How: Rethinking Retrieval-Enhanced Speculative Decodingcited by 3
  10. 2025Cacheback: Speculative Decoding With Nothing But Cachecited by 3
  11. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Modelscited by 3
  12. 2026Trees from Marginals: Autoregressive drafting with factorized priorscited by 3
  13. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  14. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2
  15. 2025Accelerated Test-Time Scaling with Model-Free Speculative Samplingcited by 2
  16. 2025SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generationcited by 2
  17. 2026JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergencecited by 2
  18. 2026Goose: Anisotropic Speculation Trees for Training-Free Speculative Decodingcited by 2
  19. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  20. 2026TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decodingcited by 2
  21. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  22. 2026ReTrace: Rejected-Trajectory Conditioning for Speculative Decodingcited by 2
  23. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  24. 2026Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verificationcited by 1
  25. 2026PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decodingcited by 1
  26. 2026AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decodingcited by 1
  27. 2026OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Callingcited by 1
  28. 2026When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agentscited by 1
  29. 2026ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shapingcited by 1
  30. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decodingcited by 1
  31. 2026Carryover Drafting: Recycling Rejected States for Speculative Decodingcited by 1
  32. 2025OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputscited by 0
  33. 2026Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computescited by 0
  34. 2026ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decodingcited by 0
  35. 2026Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  2. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  3. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  4. 2024SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens
  5. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  6. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  7. 2025HAMburger: Accelerating LLM Inference via Token Smashing
  8. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction