mlsanityData to 14 Oct 2026

Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder

arXiv 2304.04052 ↗First posted 8 Apr 2023cs.CL cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

The sequence-to-sequence (seq2seq) task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by the Encoder-Decoder framework which requires an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task. Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder architecture and the decoder-only language model framework through the analysis of a regularized encoder-decoder structure. This structure is designed to replicate all behaviors in the classical decoder-only language model but has an encoder and a decoder making it easier to be compared with the classical encoder-decoder structure. Based on the analysis, we unveil the attention degeneration problem in the language model, namely, as the generation step number grows, less and less attention is focused on the source sequence. To give a quantitative understanding of this problem, we conduct a theoretical sensitivity analysis of the attention output with respect to the source input. Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem. Experimental results on machine translation, summarization, and data-to-text generation tasks support our analysis and demonstrate the effectiveness of our proposed model.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 47 of its 73 references that are in the map. First posted on arXiv 8 Apr 2023.

Builds on

47 of 73 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
35 more
  1. 2018Deep contextualized word representationscited by 2,290
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  11. 2022Galactica: A Large Language Model for Sciencecited by 744
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  13. 2018Universal Transformerscited by 647
  14. 2017Convolutional Sequence to Sequence Learningcited by 552
  15. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  16. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  17. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  18. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  19. 2017Six Challenges for Neural Machine Translationcited by 335
  20. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  21. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  22. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  23. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  24. 2020The Lipschitz Constant of Self-Attentioncited by 174
  25. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  26. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  27. 2016Neural Summarization by Extracting Sentences and Wordscited by 116
  28. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  29. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  30. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  31. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  32. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  33. 2017Affect-LM: A Neural Language Model for Customizable Affective Text Generationcited by 50
  34. 2021Controlling Hallucinations at Word Level in Data-to-Text Generationcited by 41
  35. 2020Partially-Aligned Data-to-Text Generation with Distant Supervisioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 20; 2025: 10; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  2. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  3. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  4. 2024Theoretical limitations of multi-layer Transformercited by 45
  5. 2023Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistancecited by 33
  6. 2024A Survey of Deep Learning and Foundation Models for Time Series Forecastingcited by 33
  7. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 18
  8. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 17
  9. 2023ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Modelscited by 15
  10. 2024Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agentcited by 14
  11. 2024Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesiscited by 12
  12. 2024Decoder-only Streaming Transformer for Simultaneous Translationcited by 11
32 more
  1. 2024Denoising Autoregressive Representation Learningcited by 9
  2. 2023Customizable Combination of Parameter-Efficient Modules for Multi-Task Learningcited by 8
  3. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  4. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7
  5. 2024Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysiscited by 6
  6. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  7. 2024On the Challenges of Fuzzing Techniques via Large Language Modelscited by 4
  8. 2024TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizerscited by 4
  9. 2024Summarizing long regulatory documents with a multi-step pipelinecited by 4
  10. 2024Scaling Laws of Decoder-Only Models on the Multilingual Machine Translation Taskcited by 4
  11. 2025Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Modelcited by 4
  12. 2023MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generationcited by 3
  13. 2024Strada-LLM: Graph LLM for traffic predictioncited by 3
  14. 2025HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signalscited by 3
  15. 2023An Introduction to Natural Language Processing Techniques and Framework for Clinical Implementation in Radiation Oncologycited by 2
  16. 2023TPPoet: Transformer-Based Persian Poem Generation using Minimal Data and Advanced Decoding Techniquescited by 2
  17. 2024Efficient Encoder-Decoder Transformer Decoding for Decomposable Taskscited by 2
  18. 2024Investigating the translation capabilities of Large Language Models trained on parallel data onlycited by 2
  19. 2024Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Modelscited by 2
  20. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  21. 2025Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understandingcited by 2
  22. 2023Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precisioncited by 1
  23. 2025Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoderscited by 1
  24. 2025Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approachcited by 1
  25. 2023Reward Engineering for Generating Semi-structured Explanationcited by 0
  26. 2024CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarizationcited by 0
  27. 2025VNJPTranslate: A comprehensive pipeline for Vietnamese-Japanese translationcited by 0
  28. 2025The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecastingcited by 0
  29. 2025LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Modelscited by 0
  30. 2026Transformer-Based Approach for Automated Functional Group Replacement in Chemical Compoundscited by 0
  31. 2026Selecting Language Models for Social Science: Start Small, Start Open, and Validatecited by 0
  32. 2026Leveraging Physiological Signals to Predict Exam Outcomes with Machine Learningcited by 0