CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models
Not on the map. Why?
Abstract (arXiv)
We introduce a comprehensive continual learning dataset and benchmark (CurlL) grounded in human developmental trajectories from ages 5-10, enabling systematic and fine-grained assessment of models' ability to progressively acquire new skills. CurlL spans five developmental stages (0-4) covering ages 5-10, supported by a skill graph that breaks down broad skills into smaller abilities, concrete goals, and measurable indicators, while also capturing which abilities build on others. We generate a 23.4B-token synthetic dataset with controlled skill progression, vocabulary complexity, and format diversity, comprising paragraphs, comprehension-based QA (CQA), skill-testing QA (CSQA), and instruction-response (IR) pairs. Stage-wise token counts range from 2.12B to 6.78B tokens, supporting precise analysis of forgetting, forward transfer, and backward transfer. Using a 135M-parameter transformer trained under independent, joint, and sequential (continual) setups, we show trade-offs in skill retention and transfer efficiency. By mirroring human learning patterns and providing fine-grained control over skill dependencies, this work advances continual learning evaluations for language models.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 37 of its 105 references that are in the map. First posted on arXiv 14 Oct 2025.
Builds on
37 of 105 references.
References · 37
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 436
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 271
- 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 230
- 2023Progressive Prompts: Continual Learning for Language Modelscited by 196
- 2021Towards Continual Knowledge Learning of Language Modelscited by 173
- 2022Fine-tuned Language Models are Continual Learnerscited by 154
- 2024Continual Learning for Large Language Models: A Surveycited by 144
- 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
- 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
- 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 115
- 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
25 more
- 2025LifelongAgentBench: Evaluating LLM Agents as Lifelong Learnerscited by 71
- 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 67
- 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 63
- 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
- 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 62
- 2021Curriculum-Meta Learning for Order-Robust Continual Relation Extractioncited by 34
- 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
- 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 31
- 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
- 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 22
- 2023Is forgetting less a good inductive bias for forward transfer?cited by 18
- 2024Promises and challenges of generative artificial intelligence for human learningcited by 17
- 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
- 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 13
- 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
- 2024SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Modelscited by 11
- 2023Continual Learning with Dirichlet Generative-based Rehearsalcited by 7
- 2024Refine Large Language Model Fine-tuning via Instruction Vectorcited by 7
- 2025DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learningcited by 7
- 2023Online Continual Knowledge Learning for Language Modelscited by 6
- 2023From Static to Dynamic: A Continual Learning Framework for Large Language Modelscited by 4
- 2025Lifelong Learning with Task-Specific Adaptation: Addressing the Stability-Plasticity Dilemmacited by 4
- 2025Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuningcited by 2
- 2022Latent Properties of Lifelong Learning Systemscited by 1
- 2025Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfercited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.