mlsanityData to 14 Oct 2026

Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs

arXiv 2405.15208 ↗First posted 24 May 2024cs.CL cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Large language models have demonstrated exceptional capability in natural language understanding and generation. However, their generation speed is limited by the inherently sequential nature of their decoding process, posing challenges for real-time applications. This paper introduces Lexical Unit Decoding (LUD), a novel decoding methodology implemented in a data-driven manner, accelerating the decoding process without sacrificing output quality. The core of our approach is the observation that a pre-trained language model can confidently predict multiple contiguous tokens, forming the basis for a textitlexical unit, in which these contiguous tokens could be decoded in parallel. Extensive experiments validate that our method substantially reduces decoding time while maintaining generation quality, i.e., 33% speed up on natural language generation with no quality loss, and 30% speed up on code generation with a negligible quality loss of 3%. Distinctively, LUD requires no auxiliary models and does not require changes to existing architectures. It can also be integrated with other decoding acceleration methods, thus achieving an even more pronounced inference efficiency boost. We posit that the foundational principles of LUD could define a new decoding paradigm for future language models, enhancing their applicability for a broader spectrum of applications. All codes are be publicly available at https://github.com/tjunlp-lab/Lexical-Unit-Decoding-LUD-. Keywords: Parallel Decoding, Lexical Unit Decoding, Large Language Model

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 40 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

27 of 40 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2017Non-Autoregressive Neural Machine Translationcited by 463
  11. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
15 more
  1. 2021Scaling Laws for Transfercited by 272
  2. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  3. 2019Depth-Adaptive Transformercited by 236
  4. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  5. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  6. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 104
  7. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  8. 2021Sparse is Enough in Scaling Transformerscited by 86
  9. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 74
  10. 2020Why should we add early exits to neural networks?cited by 70
  11. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  12. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  13. 2023Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedycited by 50
  14. 2020Controlling Computation versus Quality for Neural Sequence Modelscited by 27
  15. 2023X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agentscited by 13

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Trainingcited by 5
  2. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  2. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
  3. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  4. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
  5. 2024SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens
  6. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  7. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models
  8. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling