A surprisal oracle for when every layer counts
Not on the map. Why?
Abstract (arXiv)
Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 4 Dec 2024.
Builds on
7 of 15 references.
References · 7
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
- 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
- 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
- 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
- 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
Built on by
Not cited yet.