mlsanityData to 7 Oct 2026

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training

arXiv 2406.17404 ↗First posted 25 Jun 2024cs.CL cs.LGcited by 5

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Existing speculative decoding methods typically require additional model structure and training processes to assist the model for draft token generation. This makes the migration of acceleration methods to the new model more costly and more demanding on device memory. To address this problem, we propose the Make Some Noise (MSN) training framework as a replacement for the supervised fine-tuning stage of the large language model. The training method simply introduces some noise at the input for the model to learn the denoising task. It significantly enhances the parallel decoding capability of the model without affecting the original task capability. In addition, we propose a tree-based retrieval-augmented Jacobi (TR-Jacobi) decoding strategy to further improve the inference speed of MSN models. Experiments in both the general and code domains have shown that MSN can improve inference speed by 2.3-2.7x times without compromising model performance. The MSN model also achieves comparable acceleration ratios to the SOTA model with additional model structure on Spec-Bench.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 37 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

28 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs24 May
  2. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding21 Feb
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2023StarCoder: may the source be with you!cited by 790
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
7 more
  1. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  2. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  3. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  4. 2024The pitfalls of next-token predictioncited by 156
  5. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  6. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  7. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  2. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  3. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  4. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  5. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
  2. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  3. 2026Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  6. 2024SSSD: Simply-Scalable Speculative Decoding
  7. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  8. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding