# Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

arXiv: https://arxiv.org/abs/2402.10738 · HTML: https://mlsanity.com/paper/2402.10738/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://mlsanity.com/paper/1904.01608/) (2019)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [SciNLI: A Corpus for Natural Language Inference on Scientific Text](https://mlsanity.com/paper/2203.06728/) (2022)
- [CITING: Large Language Models Create Curriculum for Instruction Tuning](https://mlsanity.com/paper/2310.02527/) (2023)
- [Misconfidence-based Demonstration Selection for LLM In-Context Learning](https://mlsanity.com/paper/2401.06301/) (2024)
- [Hierarchical Curriculum Learning for AMR Parsing](https://mlsanity.com/paper/2110.07855/) (2021)

## Built on by (most cited first, 30 of 47)

- [The Best Instruction-Tuning Data are Those That Fit](https://mlsanity.com/paper/2502.04194/) (2025)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2412.07689/) (2024)
- [Octopus: On-device language model for function calling of software APIs](https://mlsanity.com/paper/2404.01549/) (2024)
- [VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning](https://mlsanity.com/paper/2507.22607/) (2025)
- [SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs](https://mlsanity.com/paper/2510.24021/) (2025)
- [Revisiting In-context Learning Inference Circuit in Large Language Models](https://mlsanity.com/paper/2410.04468/) (2024)
- [RAEmoLLM: Retrieval Augmented LLMs for Cross-Domain Misinformation Detection Using In-Context Learning Based on Emotional Information](https://mlsanity.com/paper/2406.11093/) (2024)
- [Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning](https://mlsanity.com/paper/2507.22887/) (2025)
- [MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs](https://mlsanity.com/paper/2410.13502/) (2024)
- [Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning](https://mlsanity.com/paper/2502.15401/) (2025)
- [IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark](https://mlsanity.com/paper/2507.14449/) (2025)
- [What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning](https://mlsanity.com/paper/2510.19099/) (2025)
- [History-Aware Reasoning for GUI Agents](https://mlsanity.com/paper/2511.09127/) (2025)
- [Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning](https://mlsanity.com/paper/2405.01649/) (2024)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks](https://mlsanity.com/paper/2507.01321/) (2025)
- [Revisiting In-Context Learning with Long Context Language Models](https://mlsanity.com/paper/2412.16926/) (2024)
- [The Few-shot Dilemma: Over-prompting Large Language Models](https://mlsanity.com/paper/2509.13196/) (2025)
- [Does Few-Shot Learning Help LLM Performance in Code Synthesis?](https://mlsanity.com/paper/2412.02906/) (2024)
- [OptiSeq: Ordering Examples On-The-Fly for In-Context Learning](https://mlsanity.com/paper/2501.15030/) (2025)
- [From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models](https://mlsanity.com/paper/2605.20177/) (2026)
- [PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks](https://mlsanity.com/paper/2412.05710/) (2024)
- [DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer](https://mlsanity.com/paper/2507.23554/) (2025)
- [DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulation](https://mlsanity.com/paper/2505.14163/) (2025)
- [Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation](https://mlsanity.com/paper/2503.22776/) (2025)
- [Reasoning Steps as Curriculum: Using Depth of Thought as a Difficulty Signal for Tuning LLMs](https://mlsanity.com/paper/2508.18279/) (2025)
- [On Theoretical Interpretations of Concept-Based In-Context Learning](https://mlsanity.com/paper/2509.20882/) (2025)
- [StaICC: Standardized Evaluation for Classification Task in In-context Learning](https://mlsanity.com/paper/2501.15708/) (2025)

## Abstract (arXiv)

Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require high computational costs to introduce the priori knowledge. In this paper, inspired by the human learning process, we propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL). The ICCL implies gradually increasing the complexity of prompt demonstrations during the inference process. The difficulty can be assessed by human experts or LLMs-driven metrics, such as perplexity. Then we design extensive experiments to discuss the effectiveness of the ICCL at both corpus-level and instance-level. Moreover, we also investigate the formation mechanism of LLM's ICCL capability. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for representative open-source LLMs. To facilitate further research and applications by other scholars, we make the code publicly available.