mlsanityData to 14 Oct 2026

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

arXiv 2609.17241 ↗First posted 15 Sep 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop performs authoritative full-model verification through a state-reuse mechanism. Crucially, the outer loop also utilizes final-layer bonus logits to correct existing paths and supplement the tree with high-confidence candidates for subsequent cycles. Experimental results across diverse benchmarks demonstrate that ECHO significantly boosts mean accepted tokens and achieves a 2.4× to 2.9× speedup, outperforming existing state-of-the-art baselines with negligible engineering overhead and no extra deployment parameters, albeit with a one-shot fine-tuning dependency for optimal acceleration. The code is available at https://github.com/whucs21Mzy/ECHO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 37 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

28 of 37 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding16 Apr
  2. 2025HiSpec: Hierarchical Speculative Decoding for LLMs1 Oct
  3. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation2 Jul
  4. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  5. 2024SAM Decoding: Speculative Decoding via Suffix Automaton16 Nov
  6. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  8. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
2 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  9. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 47
  12. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
2 more
  1. 2025SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layerscited by 9
  2. 2025Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  2. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
  3. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  4. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding
  6. 2026FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
  7. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  8. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding