mlsanityData to 14 Oct 2026

Accelerating Transformer Inference for Translation via Parallel Decoding

arXiv 2305.10427 ↗First posted 17 May 2023cs.CL cs.AI cs.LGcited by 124

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 70 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

44 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  3. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2016Sequence-Level Knowledge Distillationcited by 801
29 more
  1. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  5. 2017Non-Autoregressive Neural Machine Translationcited by 463
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
  10. 2021Gender Bias in Machine Translationcited by 117
  11. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  12. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  13. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  14. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  15. 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
  16. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 75
  17. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translationcited by 65
  18. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  19. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  20. 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 52
  21. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  22. 2018Semi-Autoregressive Neural Machine Translationcited by 48
  23. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  24. 2020Semi-Autoregressive Training Improves Mask-Predict Decodingcited by 33
  25. 2021AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translatecited by 24
  26. 2020Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translationcited by 16
  27. 2017Fast Generation for Convolutional Autoregressive Modelscited by 14
  28. 2023Latent Autoregressive Source Separationcited by 12
  29. 2022Answer Fast: Accelerating BERT on the Tensor Streaming Processorcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 48; 2025: 44; 2026: 24.

124 papers.

Sort:

By area:

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  2. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  7. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  8. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  9. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  10. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  11. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  12. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
38 more
  1. 2024CLLMs: Consistency Large Language Modelscited by 72
  2. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  3. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  4. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 66
  5. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  6. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  7. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  8. 2025CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decodingcited by 38
  9. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  10. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 37
  11. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 37
  12. 2024Decoding Speculative Decodingcited by 34
  13. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
  14. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  15. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 29
  16. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 28
  17. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 26
  18. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  19. 2024On Speculative Decoding for Multimodal Large Language Modelscited by 25
  20. 2025Parallel Continuous Chain-of-Thought with Jacobi Iterationcited by 25
  21. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  22. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  23. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  24. 2024ZipAR: Parallel Auto-regressive Image Generation through Spatial Localitycited by 21
  25. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 21
  26. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 19
  27. 2024Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matchingcited by 18
  28. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifactscited by 17
  29. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  30. 2025Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devicescited by 15
  31. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 15
  32. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 14
  33. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  34. 2023Camoscio: an Italian Instruction-tuned LLaMAcited by 13
  35. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  36. 2024Remote Timing Attacks on Efficient Language Model Inferencecited by 13
  37. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  38. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 13

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  2. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
  3. 2024Tandem Transformers for Inference Efficient LLMs
  4. 2024Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
  5. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  6. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translation
  7. 2025Parallel Token Prediction for Language Models
  8. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models