mlsanityData to 7 Oct 2026

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

arXiv 2601.03448 ↗First posted 6 Jan 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 82 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

41 of 82 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
29 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  3. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  5. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  6. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  7. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  10. 2024Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligencecited by 59
  11. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  12. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  13. 2020Syntax-Enhanced Pre-trained Modelcited by 29
  14. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  15. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  16. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  17. 2020On Losses for Modern Language Modelscited by 21
  18. 2021Frustratingly Simple Pretraining Alternatives to Masked Language Modelingcited by 17
  19. 2022Exploring the Role of Task Transferability in Large-Scale Multi-Task Learningcited by 17
  20. 2022How does the pre-training objective affect what large language models learn about linguistic properties?cited by 16
  21. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  22. 2023SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERTcited by 15
  23. 2023What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structurecited by 13
  24. 2024Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Trainingcited by 11
  25. 2019Scalable Syntax-Aware Language Models Using Knowledge Distillationcited by 8
  26. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  27. 2023How does the task complexity of masked pretraining objectives affect downstream performance?cited by 2
  28. 2023Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spacescited by 2
  29. 2023Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  2. 2026Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibilitycited by 2
  3. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  4. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0