mlsanityData to 7 Oct 2026

ConFu: Contemplate the Future for Better Speculative Sampling

arXiv 2603.08899 ↗First posted 9 Mar 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose ConFu (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate tokens and soft prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor token sampling and future prediction replication that learns robust future prediction. ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8--11% on Llama-3 3B/8B and by approximately 20% on Qwen-3 4B across downstream tasks. We believe our work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 29 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

26 of 29 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decoding16 Feb
  3. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  4. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference12 Jul
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs29 Feb
  7. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
1 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  9. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  10. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  11. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  12. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
1 more
  1. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2024Decoding Speculative Decoding
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  6. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  7. 2026Faster LLM Inference via Sequential Monte Carlo
  8. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference