mlsanityData to 7 Oct 2026

A Natural Bias for Language Generation Models

arXiv 2212.09686 ↗First posted 19 Dec 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus. The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity? Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model's final linear layer with the log-unigram distribution. We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 55 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

29 of 55 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  10. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  11. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  12. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
17 more
  1. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  2. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  3. 2021Language Model Evaluation Beyond Perplexitycited by 77
  4. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  5. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  6. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  7. 2021Word Acquisition in Neural Language Modelscited by 41
  8. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  9. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  10. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  11. 2019Evaluating Computational Language Models with Scaling Properties of Natural Languagecited by 20
  12. 2017Do Neural Nets Learn Statistical Laws behind Natural Language?cited by 16
  13. 2014Compositional Morphology for Word Representations and Language Modellingcited by 11
  14. 2016Generalizing and Hybridizing Count-based and Neural Language Modelscited by 10
  15. 2021Modeling the Unigram Distributioncited by 7
  16. 2018Parameter Re-Initialization through Cyclical Batch Size Schedulescited by 3
  17. 2022N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 1.

6 papers.

  1. 2024Confidence Regulation Neurons in Language Modelscited by 57
  2. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  3. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  4. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  5. 2024Fantastic Biases (What are They) and Where to Find Themcited by 0
  6. 2026The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priorscited by 0