mlsanityData to 7 Oct 2026

Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

arXiv 2203.10326 ↗First posted 19 Mar 2022cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language. A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language. Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 19 Mar 2022.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2019How multilingual is Multilingual BERT?cited by 642
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
20 more
  1. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  2. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  3. 2016Using the Output Embedding to Improve Language Modelscited by 266
  4. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  5. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  9. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  10. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  11. 2020UnNatural Language Inferencecited by 68
  12. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  13. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  14. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  15. 2021Examining the Inductive Bias of Neural Language Models with Artificial Languagescited by 40
  16. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  17. 2015Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representationcited by 28
  18. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  19. 2018LSTMs Exploit Linguistic Attributes of Datacited by 3
  20. 2020Revisiting the Context Window for Cross-lingual Word Embeddingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 2; 2025: 3; 2026: 8.

28 papers.

Sort:

By area:

  1. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  2. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  3. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  4. 2023Gloss-Free End-to-End Sign Language Translationcited by 31
  5. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  6. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  7. 2022Pre-training Vision Transformers with Formula-driven Supervised Learningcited by 21
  8. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11
  9. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10
  10. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  11. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  12. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
16 more
  1. 2023Second Language Acquisition of Neural Language Modelscited by 8
  2. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  3. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learningcited by 7
  4. 2023Does Deep Learning Learn to Abstract? A Systematic Probing Frameworkcited by 6
  5. 2024Measuring Cross-lingual Transfer in Bytescited by 6
  6. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  7. 2025Transfer of Structural Knowledge from Synthetic Languagescited by 3
  8. 2023Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?cited by 2
  9. 2022MonoByte: A Pool of Monolingual Byte-level Language Modelscited by 1
  10. 2024Can Language Models Induce Grammatical Knowledge from Indirect Evidence?cited by 1
  11. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  12. 2026Language Acquisition Device in Large Language Modelscited by 1
  13. 2026Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequencescited by 0
  14. 2026Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languagescited by 0
  15. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0
  16. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0