Token-Level Fitting Issues of Seq2seq Models
Not on the map. Why?
Abstract (arXiv)
Sequence-to-sequence (seq2seq) models have been widely used for natural language processing, computer vision, and other deep learning tasks. We find that seq2seq models trained with early-stopping suffer from issues at the token level. In particular, while some tokens in the vocabulary demonstrate overfitting, others underfit when training is stopped. Experiments show that the phenomena are pervasive in different models, even in fine-tuned large pretrained-models. We identify three major factors that influence token-level fitting, which include token frequency, parts-of-speech, and prediction discrepancy. Further, we find that external factors such as language, model size, domain, data scale, and pretraining can also influence the fitting of tokens.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 8 May 2023.
Builds on
22 of 36 references.
References · 22
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2017A Closer Look at Memorization in Deep Networkscited by 936
- 2020Overfitting in adversarially robust deep learningcited by 504
- 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
- 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
10 more
- 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
- 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
- 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
- 2021Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Modelscited by 39
- 2021Sequence Length is a Domain: Length-based Overfitting in Transformer Modelscited by 34
- 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
- 2022On the Generalization Mystery in Deep Learningcited by 25
- 2020On Long-Tailed Phenomena in Neural Machine Translationcited by 20
- 2020Dynamic Curriculum Learning for Low-Resource Neural Machine Translationcited by 11
- 2017Complex Structure Leads to Overfitting: A Structure Regularization Decoding Method for Natural Language Processingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.