# Lossless Acceleration for Seq2seq Generation with Aggressive Decoding arXiv: https://arxiv.org/abs/2205.10350 ยท HTML: https://mlsanity.com/paper/2205.10350/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 54 references that are in the map. - First posted on arXiv 20 May 2022. ## Builds on (most cited first, 30 of 36) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018) - [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017) - [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022) - [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020) - [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018) - [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020) - [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020) - [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018) - [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019) - [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018) - [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020) - [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020) - [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021) - [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020) - [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020) - [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019) - [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018) - [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020) - [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020) - [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021) - [Semi-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1808.08583/) (2018) - [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019) - [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021) - [Semi-Autoregressive Training Improves Mask-Predict Decoding](https://mlsanity.com/paper/2001.08785/) (2020) - [Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction](https://mlsanity.com/paper/2010.03260/) (2020) ## Built on by (most cited first, 18 of 18) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024) - [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023) - [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023) - [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023) - [Inference with Reference: Lossless Acceleration of Large Language Models](https://mlsanity.com/paper/2304.04487/) (2023) - [Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy](https://mlsanity.com/paper/2312.12728/) (2023) - [EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start](https://mlsanity.com/paper/2205.12209/) (2022) - [Meet in the Middle: A New Pre-training Paradigm](https://mlsanity.com/paper/2303.07295/) (2023) - [Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits](https://mlsanity.com/paper/2410.18234/) (2024) - [Steering Pretrained Drafters during Speculative Decoding](https://mlsanity.com/paper/2511.09844/) (2025) - [RecycleGPT: An Autoregressive Language Model with Recyclable Module](https://mlsanity.com/paper/2308.03421/) (2023) - [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024) - [RedPenNet for Grammatical Error Correction: Outputs to Tokens, Attentions to Spans](https://mlsanity.com/paper/2309.10898/) (2023) - [Confidence-Modulated Speculative Decoding for Large Language Models](https://mlsanity.com/paper/2508.15371/) (2025) - [Switchable Decision: Dynamic Neural Generation Networks](https://mlsanity.com/paper/2405.04513/) (2024) - [TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding](https://mlsanity.com/paper/2606.08184/) (2026) - [Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection](https://mlsanity.com/paper/2607.05148/) (2026) ## Abstract (arXiv) We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding. Unlike the previous efforts (e.g., non-autoregressive decoding) speeding up seq2seq generation at the cost of quality loss, our approach aims to yield the identical (or better) generation compared with autoregressive decoding but in a significant speedup, achieved by innovative cooperation of aggressive decoding and verification that are both efficient due to parallel computing. We propose two Aggressive Decoding paradigms for 2 kinds of seq2seq tasks: 1) For the seq2seq tasks whose inputs and outputs are highly similar (e.g., Grammatical Error Correction), we propose Input-guided Aggressive Decoding (IAD) that aggressively copies from the input sentence as drafted decoded tokens to verify in parallel; 2) For other general seq2seq tasks (e.g., Machine Translation), we propose Generalized Aggressive Decoding (GAD) that first employs an additional non-autoregressive decoding model for aggressive decoding and then verifies in parallel in the autoregressive manner. We test Aggressive Decoding on the most popular 6-layer Transformer model on GPU in multiple seq2seq tasks: 1) For IAD, we show that it can introduce a 7x-9x speedup for the Transformer in Grammatical Error Correction and Text Simplification tasks with the identical results as greedy decoding; 2) For GAD, we observe a 3x-5x speedup with the identical or even better quality in two important seq2seq tasks: Machine Translation and Abstractive Summarization. Moreover, Aggressive Decoding can benefit even more from stronger computing devices that are better at parallel computing. Given the lossless quality as well as significant and promising speedup, we believe Aggressive Decoding may potentially evolve into a de facto standard for efficient and lossless seq2seq generation in the near future.