mlsanityData to 7 Oct 2026

ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

arXiv 2310.11282 ↗First posted 17 Oct 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We present the submission of the ILLC at the University of Amsterdam to the BabyLM challenge (Warstadt et al., 2023), in the strict-small track. Our final model, ChapGTP, is a masked language model that was trained for 200 epochs, aided by a novel data augmentation technique called Automatic Task Formation. We discuss in detail the performance of this model on the three evaluation suites: BLiMP, (Super)GLUE, and MSGS. Furthermore, we present a wide range of methods that were ultimately not included in the model, but may serve as inspiration for training LMs in low-resource settings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

43 of 68 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
31 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019Green AIcited by 620
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  6. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  9. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  10. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  11. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 123
  12. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  13. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  14. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  15. 2017Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialogcited by 86
  16. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  17. 2020The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Researchcited by 67
  18. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 65
  19. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  20. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  21. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  22. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  23. 2018How agents see things: On visual representations in an emergent language gamecited by 45
  24. 2020Transferring Inductive Biases through Knowledge Distillationcited by 41
  25. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  26. 2020Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answeringcited by 39
  27. 2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19
  28. 2022Linking Emergent and Natural Languages via Corpus Transfercited by 19
  29. 2023Putting Natural in Natural Language Processingcited by 17
  30. 2020Internal and external pressures on language emergence: least effort, object constancy and frequencycited by 13
  31. 2023Quantifying the perceptual value of lexical and non-lexical channels in speechcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 2
  3. 2024Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenizationcited by 0