mlsanityData to 7 Oct 2026

Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning

arXiv 2405.07490 ↗First posted 13 May 2024cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has improved text understanding and generation but poses challenges in computational resources. This study proposes a curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Experiments with Mistral-7B (Jiang et al., 2023) and Gemma-7B (Team et al., 2024) models demonstrate that curriculum learning slightly improves performance compared to traditional random data shuffling. Notably, we observed that sorting data based on our proposed attention criteria generally led to better performance. This approach offers a sustainable method to enhance LLM performance without increasing model size or dataset volume, addressing scalability challenges in LLM training.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

15 of 24 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  11. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  12. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
3 more
  1. 2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
  2. 2019Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narrativescited by 59
  3. 2021Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learningcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  3. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  4. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  5. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  6. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 10
  7. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  8. 2025Data Efficacy for Language Model Trainingcited by 8
  9. 2025LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agentscited by 6
  10. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 5
  11. 2025A Survey of LLM × DATAcited by 4
  12. 2025Complexity-aware fine-tuningcited by 2
16 more
  1. 2026EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuningcited by 2
  2. 2026Small-Margin Preferences Still Matter-If You Train Them Rightcited by 2
  3. 2026Progressive Multi-Agent Reasoning for Biological Perturbation Predictioncited by 2
  4. 2026Demystifying Data Organization for Enhanced LLM Trainingcited by 2
  5. 2025Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2cited by 1
  6. 2025Small Models, Smarter Learning: The Power of Joint Task Trainingcited by 1
  7. 2025Logical Structure as Knowledge: Enhancing LLM Reasoning via Structured Logical Knowledge Density Estimationcited by 1
  8. 2026Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalizationcited by 1
  9. 2025PMG: Progressive Motion Generation via Sparse Anchor Postures Curriculum Learningcited by 0
  10. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0
  11. 2025Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Datacited by 0
  12. 2025General Intelligence-based Fragmentation (GIF): A framework for peak-labeled spectra simulationcited by 0
  13. 2025From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Taskscited by 0
  14. 2026D3: Dynamic Directional Graph-Constrained Data Scheduling for LLM Trainingcited by 0
  15. 2026Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalizationcited by 0
  16. 2026Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamicscited by 0