mlsanityData to 14 Oct 2026

Instruction Tuning with Human Curriculum

arXiv 2310.09518 ↗First posted 14 Oct 2023cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In this work, we (1) introduce Curriculum Instruction Tuning, (2) explore the potential advantages of employing diverse curriculum strategies, and (3) delineate a synthetic instruction-response generation framework that complements our theoretical approach. Distinct from the existing instruction tuning dataset, our generation pipeline is systematically structured to emulate the sequential and orderly characteristic of human learning. Additionally, we describe a methodology for generating instruction-response datasets that extensively span the various stages of human education, from middle school through the graduate level, utilizing educational subject catalogs. Before training, we meticulously organize the instruction data to ensure that questions escalate in difficulty regarding (A) the subject matter and (B) the intricacy of the instructions. The findings of our study reveal that substantial improvements in performance can be achieved through the mere application of curriculum ordering to instruction data (achieving gains of +4.76 on TruthfulQA, +2.98 on MMLU, +2.8 on OpenbookQA, and +1.28 on ARC-hard) compared to random shuffling. This enhancement is achieved without incurring additional computational expenses. Through comprehensive experimentation, we observe that the advantages of our proposed method are consistently evident across nine benchmarks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 37 of its 93 references that are in the map. First posted on arXiv 14 Oct 2023.

Builds on

37 of 93 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
25 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023LIMA: Less Is More for Alignmentcited by 1,100
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  8. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  11. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  13. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  14. 2023Poisoning Language Models During Instruction Tuningcited by 269
  15. 2022Language models show human-like content effects on reasoning taskscited by 173
  16. 2020When Do Curricula Work?cited by 106
  17. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 65
  18. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  19. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
  20. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 49
  21. 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
  22. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  23. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18
  24. 2023Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Explorationcited by 18
  25. 2023CITING: Large Language Models Create Curriculum for Instruction Tuningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  2. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  3. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  4. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  5. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  6. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  7. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  8. 2026Self-Distilled Policy Gradientcited by 8
  9. 2026NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harnesscited by 4
  10. 2025ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuningcited by 3
  11. 2026RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Modelscited by 3
  12. 2025From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learningcited by 2
3 more
  1. 2024Building an Ethical and Trustworthy Biomedical AI Ecosystem for the Translational and Clinical Integration of Foundational Modelscited by 0
  2. 2025Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Frameworkcited by 0
  3. 2026OmniEdu: Open Foundation Models for Learning and Teachingcited by 0