# Curriculum Learning for Small Code Language Models arXiv: https://arxiv.org/abs/2407.10194 ยท HTML: https://mlsanity.com/paper/2407.10194/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 30 references that are in the map. - First posted on arXiv 14 Jul 2024. ## Builds on (most cited first, 17 of 17) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022) - [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023) - [Shortformer: Better Language Modeling using Shorter Inputs](https://mlsanity.com/paper/2012.15832/) (2020) - [The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models](https://mlsanity.com/paper/2108.06084/) (2021) - [Exploring Software Naturalness through Neural Language Models](https://mlsanity.com/paper/2006.12641/) (2020) - [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021) - [Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale](https://mlsanity.com/paper/2305.17266/) (2023) ## Built on by (most cited first, 22 of 22) - [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025) - [SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning](https://mlsanity.com/paper/2506.09016/) (2025) - [DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model](https://mlsanity.com/paper/2502.15832/) (2025) - [Mid-Training of Large Language Models: A Survey](https://mlsanity.com/paper/2510.06826/) (2025) - [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025) - [Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need](https://mlsanity.com/paper/2507.13966/) (2025) - [CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2602.03048/) (2026) - [Revisiting Multilingual Data Mixtures in Language Model Pretraining](https://mlsanity.com/paper/2510.25947/) (2025) - [Fast Quiet-STaR: Thinking Without Thought Tokens](https://mlsanity.com/paper/2505.17746/) (2025) - [Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding](https://mlsanity.com/paper/2507.09758/) (2025) - [Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks](https://mlsanity.com/paper/2502.03806/) (2025) - [DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.10493/) (2025) - [TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models](https://mlsanity.com/paper/2602.15449/) (2026) - [Smooth Scaling Laws Hide Stepwise Token Learning](https://mlsanity.com/paper/2606.29858/) (2026) - [SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding](https://mlsanity.com/paper/2505.04723/) (2025) - [Estimating the Effects of Sample Training Orders for Large Language Models without Retraining](https://mlsanity.com/paper/2505.22042/) (2025) - [Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining](https://mlsanity.com/paper/2509.24356/) (2025) - [Bandit Guided Submodular Curriculum for Adaptive Subset Selection](https://mlsanity.com/paper/2511.22944/) (2025) - [Learning to Solve Complex Problems via Dataset Decomposition](https://mlsanity.com/paper/2602.20296/) (2026) - [Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning](https://mlsanity.com/paper/2605.00433/) (2026) - [FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation](https://mlsanity.com/paper/2607.19354/) (2026) - [DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning](https://mlsanity.com/paper/2607.26457/) (2026) ## Abstract (arXiv) Code language models have emerged as useful tools for various programming tasks, yet they often struggle when it comes to complex ones. In this paper, we explore the potential of curriculum learning in enhancing the performance of these models. While prior research has suggested that curriculum learning does not necessarily help in improving the performance of language models, our results surprisingly show that this may not be the case for code language models. We demonstrate that a well-designed curriculum learning approach significantly improves the accuracy of small decoder-only code language models on the task of code execution, while its effect on code completion is less significant. To explore the potential of curriculum learning, we train multiple GPT models with 1 million parameters each to predict the next token and evaluate them on code completion and execution tasks. Our contributions include proposing a novel code difficulty assessment metric by combining software code measures, investigating the effectiveness of Curriculum Learning for code language models, and introducing a Novel Curriculum Learning schedule that enhances the performance of small decoder-only language models in code execution tasks. The results of this paper open the door for more research on the use of curriculum learning for code language models.