mlsanityData to 7 Oct 2026

Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation

arXiv 2106.06125 ↗First posted 11 Jun 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

A well-known limitation in pretrain-finetune paradigm lies in its inflexibility caused by the one-size-fits-all vocabulary. This potentially weakens the effect when applying pretrained models into natural language generation (NLG) tasks, especially for the subword distributions between upstream and downstream tasks with significant discrepancy. Towards approaching this problem, we extend the vanilla pretrain-finetune pipeline with an extra embedding transfer step. Specifically, a plug-and-play embedding generator is introduced to produce the representation of any input token, according to pre-trained embeddings of its morphologically similar ones. Thus, embeddings of mismatch tokens in downstream tasks can also be efficiently initialized. We conduct experiments on a variety of NLG tasks under the pretrain-finetune fashion. Experimental results and extensive analyses show that the proposed strategy offers us opportunities to feel free to transfer the vocabulary, leading to more efficient and better performed downstream NLG models.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

31 of 49 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
19 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  5. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  6. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  7. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  8. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  9. 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
  10. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  11. 2018Neural Abstractive Text Summarization with Sequence-to-Sequence Modelscited by 39
  12. 2019An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Modelscited by 39
  13. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  14. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  15. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  16. 2020Self-Paced Learning for Neural Machine Translationcited by 27
  17. 2021Improving Tree-Structured Decoder Training for Code Generation via Mutual Learningcited by 10
  18. 2021Exploring Dynamic Selection of Branch Expansion Orders for Code Generationcited by 10
  19. 2018Generalizing Word Embeddings using Bag of Subwordscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 2; 2024: 4; 2025: 1; 2026: 1.

14 papers.

Sort:

  1. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  2. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  3. 2022Searching for Optimal Subword Tokenization in Cross-domain NERcited by 17
  4. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  5. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  6. 2021Frequency-Aware Contrastive Learning for Neural Machine Translationcited by 11
  7. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 11
  8. 2022Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understandingcited by 6
  9. 2022ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translationcited by 4
  10. 2024Chumor 2.0: Towards Benchmarking Chinese Humor Understandingcited by 4
  11. 2024Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Bacited by 3
  12. 2023Enhanced Simultaneous Machine Translation with Word-level Policiescited by 2
2 more
  1. 2024A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Modelscited by 2
  2. 2026LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptationcited by 0