mlsanityData to 14 Oct 2026

Transfer of Structural Knowledge from Synthetic Languages

arXiv 2505.15769 ↗First posted 21 May 2025cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

19 of 32 references.

References · 19

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  7. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  8. 2020What is being transferred in transfer learning?cited by 368
  9. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  10. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  11. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  12. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
7 more
  1. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  2. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 35
  3. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  4. 2020Universal linguistic inductive biases via meta-learningcited by 29
  5. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  6. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  7. 2023SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1
  2. 2026Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languagescited by 0
  3. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0