CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
Not on the map. Why?
Abstract (arXiv)
This work investigates whether small-scale LMs can benefit from instruction tuning. We compare conversational and question-answering instruction tuning datasets, applied either in a merged or sequential curriculum, using decoder-only models with 100M and 140M parameters. Evaluation spans both fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and psycholinguistic correlation) settings. Results show that instruction tuning yields small but consistent gains in fine-tuning scenarios, with sequential curricula outperforming merged data; however, improvements do not consistently transfer to zero-shot tasks, suggesting a trade-off between interaction-focused adaptation and broad linguistic generalization. These results highlight both the potential and the constraints of adapting human-inspired learning strategies to low-resource LMs, and point toward hybrid, curriculum-based approaches for enhancing generalization under ecological training limits.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 25 references that are in the map. First posted on arXiv 29 Oct 2025.
Builds on
6 of 25 references.
References · 6
- 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
- 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
- 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Modelscited by 9
- 2023BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?cited by 7
Built on by
Not cited yet.