mlsanityData to 13 Oct 2026

Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources

arXiv 2311.09732 ↗First posted 16 Nov 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) have established the new paradigm in the field of NLP. For more powerful PLMs, one of the most popular and successful way is to continuously scale up sizes of the models and the pre-training corpora. These large corpora are generally obtained by converging smaller ones from multiple sources, they are thus growing increasingly diverse. However, the side-effects of these colossal converged corpora remain understudied. In this paper, we identify the disadvantage of heterogeneous corpora from multiple sources for pre-training PLMs. Towards coordinated pre-training on diverse corpora, we further propose source prompts (SP), which explicitly prompt the model of the data source at the pre-training and fine-tuning stages. Results of extensive experiments demonstrate that PLMs pre-trained with SP on diverse corpora gain significant improvement in various downstream tasks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,942
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
29 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  3. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 757
  4. 2023Baichuan 2: Open Large-scale Language Modelscited by 668
  5. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 532
  6. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 523
  7. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  8. 2023Data Selection for Language Models via Importance Resamplingcited by 326
  9. 2022UL2: Unifying Language Learning Paradigmscited by 294
  10. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  11. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 273
  12. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  13. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 234
  14. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  15. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  16. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  17. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  18. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  19. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  20. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  21. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  22. 2020CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Modelcited by 36
  23. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  24. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  25. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  26. 2020Transformer Based Multi-Source Domain Adaptationcited by 23
  27. 2019Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixingcited by 14
  28. 2021The Trade-offs of Domain Adaptation for Neural Language Modelscited by 14
  29. 2019Maximizing Stylistic Control and Semantic Accuracy in NLG: Personality Variation and Discourse Contrastcited by 12

Built on by

Not cited yet.