mlsanityData to 14 Oct 2026

Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing

arXiv 2604.18170 ↗First posted 20 Apr 2026cs.CL cs.AIcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

LLMs edit text and code by autoregressively regenerating the full output, even when most tokens appear verbatim in the input. We study Copy-as-Decode, a decoding-layer mechanism that recasts edit generation as structured decoding over a two-primitive grammar: <copy lines="i-j"/> references an input line range, <gen>...</gen> emits new content. A token-level FSM guarantees syntactic validity, and a serving-layer primitive updates the KV cache for each copy span via a single parallel-prefill forward rather than N autoregressive steps -- sharing the parallel-forward kernel of speculative decoding but with input tokens as the draft and program-enforced acceptance replacing probabilistic verification. We report an upper-bound analysis that requires no end-to-end training. (i) Kernel speedup: on Qwen2.5-1.5B, 7B, copying N tokens via parallel prefill is 6.8×--303× faster than autoregressive (N ∈ [8, 512], A100 80GB bf16). (ii) Copy ceiling: on ProbeEdit and HumanEvalPack-Fix (Py/JS), 74--98% of gold tokens are reachable under the line-level primitive; composed with the empirical kernel over each corpus's span histogram this yields a closed-form wall-clock bound of 29.0× / 3.4× / 4.2× (13.0× pooled). A token-level extension reaches 91--99% coverage with 4.5×--6.5× floors. (iii) Pipeline losslessness: oracle programs round-trip through the deterministic resolver on all 482 cases, localizing any downstream failure to span selection rather than the mechanism. A perturbation study shows pooled EM drops from 100% to 15.48% under off-by-one noise. A fine-tuning pilot on Qwen2.5-Coder-1.5B lifts HEvalFix-Py EM from 0/33 (untrained) to 12--17%, a learnability signal, not a production selector. Batched-serving integration and multi-file coverage are scoped as follow-up.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 29 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

25 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2024Multi-Candidate Speculative Decoding12 Jan
  5. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  7. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  9. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 16

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  4. 2015Pointer Networkscited by 895
  5. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  7. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 218
  8. 2023Efficient Guided Generation for Large Language Modelscited by 200
  9. 2019Levenshtein Transformercited by 199
  10. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  11. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  12. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
4 more
  1. 2019Encode, Tag, Realize: High-Precision Text Editingcited by 99
  2. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 75
  3. 2020Felix: Flexible Text Editing Through Tagging and Insertioncited by 58
  4. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corporacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
  2. 2024Let the Code LLM Edit Itself When You Edit the Code
  3. 2024ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding
  4. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  5. 2026DEdit: Iterative Draft Editing for Speculative Decoding
  6. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  7. 2026Edit-Based Refinement for Parallel Masked Diffusion Language Models
  8. 2023PaSS: Parallel Speculative Sampling