Pipelined Decoder for Efficient Context-Aware Text Generation
Not on the map. Why?
Abstract (arXiv)
As the basis of generative AI, an autoregressive model requires the generation of a new token depending on all the previously generated tokens, which brings high quality but also restricts the model to generate tokens one by one, forming a bottleneck limiting the generation speed. In this paper, we propose a new decoder architecture that efficiently generates text in parallel for context-aware generation tasks. Our proposed pipelined decoder initiates the generation of multiple subsequences simultaneously, and, at each time-step, it generates a new token for each subsequence to realize parallelism. Experiments on multiple text generation tasks, including question answering, text summarization, and keyphrase generation, show that our pipelined decoder significantly improves the generation speed without a significant loss of generation quality or additional memory consumption.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 29 Jun 2025.
Builds on
29 of 44 references.
References · 29
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2020Reformer: The Efficient Transformercited by 1,668
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2015Teaching Machines to Read and Comprehendcited by 1,049
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2017Non-Autoregressive Neural Machine Translationcited by 463
- 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
17 more
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
- 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 234
- 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
- 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
- 2017Deep Keyphrase Generationcited by 102
- 2020Non-Autoregressive Machine Translation with Latent Alignmentscited by 96
- 2019KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documentscited by 41
- 2019Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translationcited by 39
- 2020Exclusive Hierarchical Decoding for Deep Keyphrase Generationcited by 38
- 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
- 2019Joint Learning of Answer Selection and Answer Summary Generation in Community Question Answeringcited by 25
- 2024APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decodingcited by 24
- 2021Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generationcited by 17
- 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
- 2022Clues Before Answers: Generation-Enhanced Multiple-Choice QAcited by 9
- 2020Cascaded Text Generation with Markov Transformerscited by 7
- 2023Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answeringcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.