mlsanityData to 7 Oct 2026

Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

arXiv 2405.18400 ↗First posted 28 May 2024cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Many applications today provide users with multiple auto-complete drafts as they type, including GitHub's code completion, Gmail's smart compose, and Apple's messaging auto-suggestions. Under the hood, language models support this by running an autoregressive inference pass to provide a draft. Consequently, providing k drafts to the user requires running an expensive language model k times. To alleviate the computation cost of running k inference passes, we propose Superposed Decoding, a new decoding algorithm that generates k drafts at the computation cost of one autoregressive inference pass. We achieve this by feeding a superposition of the most recent token embeddings from the k drafts as input to the next decoding step of the language model. At every inference step we combine the k drafts with the top-k tokens to get k2 new drafts and cache the k most likely options, using an n-gram interpolation with minimal compute overhead to filter out incoherent generations. Our experiments show that k drafts from Superposed Decoding are at least as coherent and factual as Nucleus Sampling and Greedy Decoding respectively, while being at least 2.44× faster for k≥3. In a compute-normalized setting, user evaluations demonstrably favor text generated by Superposed Decoding over Nucleus Sampling. Superposed Decoding can also be combined with other decoding strategies, resulting in universal coverage gains when scaling inference time compute. Code and more examples open-sourced at https://github.com/RAIVNLab/SuperposedDecoding.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

32 of 39 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
20 more
  1. 2018Hierarchical Neural Story Generationcited by 1,182
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2023Large Language Modelscited by 774
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  11. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  12. 2023Speculative Decoding with Big Little Decodercited by 163
  13. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  14. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  15. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  16. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  17. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  18. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  19. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inferencecited by 27
  20. 2023Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layercited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  2. 2025A Faster Generalized Two-Stage Approximate Top-Kcited by 0
  3. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  4. 2026Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques
  2. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
  3. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  4. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding
  5. 2026D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
  6. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  7. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  8. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding