mlsanityData to 14 Oct 2026

Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

arXiv 2609.17435 ↗First posted 15 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We submit M'eTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-language minimal-pair benchmarks as standard diagnostics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

13 of 22 references.

References · 13

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  5. 2017Word Translation Without Parallel Datacited by 487
  6. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  7. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  8. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  9. 2024GPT or BERT: why not both?cited by 27
  10. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  11. 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 13
  12. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
1 more
  1. 2025Idiom Understanding as a Tool to Measure the Dialect Gapcited by 2

Built on by

Not cited yet.