mlsanityData to 14 Oct 2026

Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

arXiv 2607.27269 ↗First posted 29 Jul 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Multi-head latent attention (MLA) is increasingly important for long-context LLM inference because compact latent states replace the growing key-value (KV) cache and reduce decoding memory traffic. Yet most capable open checkpoints use multi-head or grouped-query attention (MHA/GQA), so conversion is needed to obtain MLA's cache efficiency without retraining from scratch. Speculative decoding offers complementary acceleration, but its speedup depends on agreement between draft proposals and target verification. We find that direct MHA/GQA-to-MLA conversion can sharply reduce this agreement: low-rank factorization and RoPE handling introduce attention-function errors that may be tolerable for standalone generation but substantially lower draft-token acceptance. We therefore formulate MLA draft construction as functional reconstruction rather than cache compression. Our end-to-end (E2E) method optimizes each converted MLA attention module to reproduce the post-output-projection response of its original MHA/GQA counterpart on calibration hidden states. This converter-agnostic post-conversion procedure preserves the converted cache and inference graph and requires neither verifier logits nor verifier supervision. We evaluate 192 model-converter-backend-method-task configurations spanning four Llama/Qwen draft-target pairs, TransMLA and MHA2MLA, HF and vLLM, and four 200-prompt tasks. With a 0.5-percentage-point reporting tolerance, Functional Reconstruction materially improves acceptance in 37 of 64 matched task cells, leaves 26 practically unchanged, and materially decreases one. Code and evaluation artifacts are available at https://github.com/swyhahaha/FunctionalMLA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 26 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

19 of 26 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs
  2. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
  3. 2026FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
  4. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
  5. 2026When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
  6. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  7. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention
  8. 2026FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding