mlsanityData to 14 Oct 2026

Controlled Randomness Improves the Performance of Transformer Models

arXiv 2310.13526 ↗First posted 20 Oct 2023cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of textual data to capture the complexity and diversity of natural language. Contrasting this, in most cases, the size of the data available to solve the specific downstream task is often dwarfed by the aforementioned pre-training dataset, especially in domains where data is scarce. We introduce controlled randomness, i.e. noise, into the training process to improve fine-tuning language models and explore the performance of targeted noise in addition to the parameters of these models. We find that adding such noise can improve the performance in our two downstream tasks of joint named entity recognition and relation extraction and text summarization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

42 of 76 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2019Text Summarization with Pretrained Encoderscited by 556
30 more
  1. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  3. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  5. 2019Neural Legal Judgment Prediction in Englishcited by 209
  6. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  7. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  8. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  9. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  10. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  11. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  12. 2017Joint Extraction of Entities and Relations Based on a Novel Tagging Schemecited by 102
  13. 2020Automated Concatenation of Embeddings for Structured Predictioncited by 86
  14. 2018Joint entity recognition and relation extraction as a multi-head selection problemcited by 59
  15. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  16. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  17. 2019Joint Extraction of Entities and Relations Based on a Novel Decomposition Strategycited by 41
  18. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  19. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  20. 2022OneRel:Joint Entity and Relation Extraction with One Module in One Stepcited by 27
  21. 2019Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Textcited by 19
  22. 2021RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarizationcited by 19
  23. 2022KPI-BERT: A Joint Named Entity Recognition and Relation Extraction Model for Financial Reportscited by 19
  24. 2019End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Modelscited by 11
  25. 2019Comprehend Medical: a Named Entity Recognition and Relationship Extraction Web Servicecited by 10
  26. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  27. 2022KPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documentscited by 9
  28. 2023An Analysis of Abstractive Text Summarization Using Pre-trained Modelscited by 7
  29. 2023Informed Named Entity Recognition Decoding for Generative Language Modelscited by 6
  30. 2022An Efficient Coarse-to-Fine Facet-Aware Unsupervised Summarization Framework based on Semantic Blockscited by 4

Built on by

Not cited yet.