# Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning

arXiv: https://arxiv.org/abs/2405.07490 · HTML: https://mlsanity.com/paper/2405.07490/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)
- [Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives](https://mlsanity.com/paper/1905.10847/) (2019)
- [Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning](https://mlsanity.com/paper/2102.03554/) (2021)

## Built on by (most cited first, 28 of 28)

- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://mlsanity.com/paper/2506.11300/) (2025)
- [How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining](https://mlsanity.com/paper/2511.18903/) (2025)
- [Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning](https://mlsanity.com/paper/2502.15401/) (2025)
- [What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning](https://mlsanity.com/paper/2510.19099/) (2025)
- [Revisiting Multilingual Data Mixtures in Language Model Pretraining](https://mlsanity.com/paper/2510.25947/) (2025)
- [Data Efficacy for Language Model Training](https://mlsanity.com/paper/2506.21545/) (2025)
- [YuLan-Mini: An Open Data-efficient Language Model](https://mlsanity.com/paper/2412.17743/) (2024)
- [LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agents](https://mlsanity.com/paper/2505.21963/) (2025)
- [A Survey of LLM × DATA](https://mlsanity.com/paper/2505.18458/) (2025)
- [Demystifying Data Organization for Enhanced LLM Training](https://mlsanity.com/paper/2605.30334/) (2026)
- [Complexity-aware fine-tuning](https://mlsanity.com/paper/2506.21220/) (2025)
- [EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuning](https://mlsanity.com/paper/2601.03725/) (2026)
- [Small-Margin Preferences Still Matter-If You Train Them Right](https://mlsanity.com/paper/2602.00954/) (2026)
- [Progressive Multi-Agent Reasoning for Biological Perturbation Prediction](https://mlsanity.com/paper/2602.07408/) (2026)
- [Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2](https://mlsanity.com/paper/2505.11643/) (2025)
- [Small Models, Smarter Learning: The Power of Joint Task Training](https://mlsanity.com/paper/2505.18369/) (2025)
- [Logical Structure as Knowledge: Enhancing LLM Reasoning via Structured Logical Knowledge Density Estimation](https://mlsanity.com/paper/2509.24836/) (2025)
- [Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization](https://mlsanity.com/paper/2601.12598/) (2026)
- [PMG: Progressive Motion Generation via Sparse Anchor Postures Curriculum Learning](https://mlsanity.com/paper/2504.16722/) (2025)
- [Estimating the Effects of Sample Training Orders for Large Language Models without Retraining](https://mlsanity.com/paper/2505.22042/) (2025)
- [Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data](https://mlsanity.com/paper/2506.14704/) (2025)
- [General Intelligence-based Fragmentation (GIF): A framework for peak-labeled spectra simulation](https://mlsanity.com/paper/2511.09571/) (2025)
- [From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks](https://mlsanity.com/paper/2512.02580/) (2025)
- [D3: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training](https://mlsanity.com/paper/2605.31164/) (2026)
- [Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization](https://mlsanity.com/paper/2608.11746/) (2026)
- [Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics](https://mlsanity.com/paper/2608.17268/) (2026)

## Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has improved text understanding and generation but poses challenges in computational resources. This study proposes a curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Experiments with Mistral-7B (Jiang et al., 2023) and Gemma-7B (Team et al., 2024) models demonstrate that curriculum learning slightly improves performance compared to traditional random data shuffling. Notably, we observed that sorting data based on our proposed attention criteria generally led to better performance. This approach offers a sustainable method to enhance LLM performance without increasing model size or dataset volume, addressing scalability challenges in LLM training.