mlsanityData to 14 Oct 2026

Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text

arXiv 2610.19294 ↗First posted 13 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

14 of 24 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  4. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  5. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  6. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  7. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  8. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 15
  9. 2025Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)cited by 11
  10. 2023Too Much Information: Keeping Training Simple for BabyLMscited by 7
  11. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 3
  12. 2024Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Datacited by 3
2 more
  1. 2025Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMscited by 3
  2. 2025Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modelingcited by 1

Built on by

Not cited yet.