mlsanityData to 14 Oct 2026

Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks

arXiv 2403.13112 ↗First posted 19 Mar 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
26 more
  1. 2020Knowledge Distillation: A Surveycited by 1,603
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  4. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  7. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  9. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  10. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  12. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  13. 2023Do We Still Need Clinical Language Models?cited by 98
  14. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  15. 2021MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluationcited by 88
  16. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  17. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 65
  18. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  19. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  20. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  21. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  22. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  23. 2022ASSIST: Towards Label Noise-Robust Dialogue State Trackingcited by 11
  24. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  25. 2023Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?cited by 5
  26. 2023Building blocks for complex tasks: Robust generative event extraction for radiology reports under domain shiftscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026IntrAgent: An LLM Agent for Content-Grounded Information Retrieval through Literature Reviewcited by 2
  2. 2026Designing RNAs with Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Decoupled Transformer for Scalable Inference in Open-domain Question Answering
  2. 2025Training and Inference Efficiency of Encoder-Decoder Speech Models
  3. 2025Return of the Encoder: Maximizing Parameter Efficiency for SLMs
  4. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  5. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  6. 2024A Thorough Examination of Decoding Methods in the Era of LLMs
  7. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers
  8. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding