# Pre-Training a Language Model Without Human Language

arXiv: https://arxiv.org/abs/2012.11995 · HTML: https://mlsanity.com/paper/2012.11995/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2020.

## Built on by (most cited first, 14 of 14)

- [Injecting structural hints: Using language models to study inductive biases in language learning](https://mlsanity.com/paper/2304.13060/) (2023)
- [Does Pretraining for Summarization Require Knowledge Transfer?](https://mlsanity.com/paper/2109.04953/) (2021)
- [Insights into Pre-training via Simpler Synthetic Tasks](https://mlsanity.com/paper/2206.10139/) (2022)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Downstream Datasets Make Surprisingly Good Pretraining Corpora](https://mlsanity.com/paper/2209.14389/) (2022)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [On the Role of Parallel Data in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2212.10173/) (2022)
- [Synthetic Pre-Training Tasks for Neural Machine Translation](https://mlsanity.com/paper/2212.09864/) (2022)
- [Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability](https://mlsanity.com/paper/2103.07162/) (2021)
- [Measuring Cross-lingual Transfer in Bytes](https://mlsanity.com/paper/2404.08191/) (2024)
- [Compositional generalization in semantic parsing with pretrained transformers](https://mlsanity.com/paper/2109.15101/) (2021)
- [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023)
- [MonoByte: A Pool of Monolingual Byte-level Language Models](https://mlsanity.com/paper/2209.11035/) (2022)
- [Bag of Lies: Robustness in Continuous Pre-training BERT](https://mlsanity.com/paper/2406.09967/) (2024)

## Abstract (arXiv)

In this paper, we study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. To this end, we pre-train different transformer-based masked language models on several corpora with certain features, and we fine-tune those language models on GLUE benchmarks. We find that models pre-trained on unstructured data beat those trained directly from scratch on downstream tasks. Our results also show that pre-training on structured data does not always make the model acquire ability that can be transferred to natural language downstream tasks. To our great astonishment, we uncover that pre-training on certain non-human language data gives GLUE performance close to performance pre-trained on another non-English language.