mlsanityData to 14 Oct 2026

Beetle: A Bilingual Model Suite for Modelling Second-Language Processing

arXiv 2609.22633 ↗First posted 18 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled language model pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable, enabling systematic and comparable experimentation of training conditions. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s) to study multilingual pretraining and computational modelling of bilingualism and second language learning. Evaluating models on human bilingual and second language reading-time prediction and grammaticality judgement tasks, we find that staged and temporally structured curricula consistently improve alignment with language learner reading time compared to balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs. The Beetle models are well suited tools to help move computational psycholinguistics beyond its prevailing monolingual, English-centric focus toward models of human bilingual processing, to study cross-lingual learning dynamics, while supporting community-based development of controlled model families.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 78 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

40 of 78 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2019Root Mean Square Layer Normalizationcited by 1,480
  9. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  10. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  11. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  12. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
28 more
  1. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 97
  2. 2022The Geometry of Multilingual Language Model Representationscited by 95
  3. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  4. 2025Overcoming catastrophic forgetting in neural networkscited by 55
  5. 2021Lower Perplexity is Not Always Human-Likecited by 52
  6. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  7. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  8. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  9. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  10. 2024How to Compute the Probability of a Wordcited by 36
  11. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  12. 2022CEFR-Based Sentence Difficulty Annotation and Assessmentcited by 28
  13. 2017Large-scale Cloze Test Dataset Created by Teacherscited by 27
  14. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  15. 2023SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERTcited by 15
  16. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  17. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  18. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  19. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  20. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  21. 2023Second Language Acquisition of Neural Language Modelscited by 8
  22. 2025On the Acquisition of Shared Grammatical Representations in Bilingual Language Modelscited by 7
  23. 2025How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoderscited by 5
  24. 2026When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Trainingcited by 5
  25. 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
  26. 2025Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languagescited by 3
  27. 2025Pico: A Modular Framework for Hypothesis-Driven Small Language Model Researchcited by 3
  28. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3

Built on by

Not cited yet.