mlsanityData to 7 Oct 2026

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

arXiv 2402.10738 ↗First posted 16 Feb 2024cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require high computational costs to introduce the priori knowledge. In this paper, inspired by the human learning process, we propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL). The ICCL implies gradually increasing the complexity of prompt demonstrations during the inference process. The difficulty can be assessed by human experts or LLMs-driven metrics, such as perplexity. Then we design extensive experiments to discuss the effectiveness of the ICCL at both corpus-level and instance-level. Moreover, we also investigate the formation mechanism of LLM's ICCL capability. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for representative open-source LLMs. To facilitate further research and applications by other scholars, we make the code publicly available.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

16 of 27 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  8. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  9. 2023Efficient Large Language Models: A Surveycited by 183
  10. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  11. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  12. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
4 more
  1. 2022SciNLI: A Corpus for Natural Language Inference on Scientific Textcited by 27
  2. 2023CITING: Large Language Models Create Curriculum for Instruction Tuningcited by 14
  3. 2024Misconfidence-based Demonstration Selection for LLM In-Context Learningcited by 14
  4. 2021Hierarchical Curriculum Learning for AMR Parsingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 28; 2026: 7.

47 papers.

Sort:

By area:

  1. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  2. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  3. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  4. 2024Octopus: On-device language model for function calling of software APIscited by 26
  5. 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
  6. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  7. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  8. 2024RAEmoLLM: Retrieval Augmented LLMs for Cross-Domain Misinformation Detection Using In-Context Learning Based on Emotional Informationcited by 14
  9. 2025Where to show Demos in Your Prompt: A Positional Bias of In-Context Learningcited by 13
  10. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  11. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  12. 2025IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmarkcited by 11
35 more
  1. 2025History-Aware Reasoning for GUI Agentscited by 11
  2. 2024Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuningcited by 10
  3. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  4. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 10
  5. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  6. 2025ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attackscited by 9
  7. 2024Revisiting In-Context Learning with Long Context Language Modelscited by 7
  8. 2025The Few-shot Dilemma: Over-prompting Large Language Modelscited by 7
  9. 2024Does Few-Shot Learning Help LLM Performance in Code Synthesis?cited by 6
  10. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  11. 2024PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Bankscited by 5
  12. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  13. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Modelscited by 5
  14. 2025DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulationcited by 4
  15. 2025Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translationcited by 3
  16. 2025Reasoning Steps as Curriculum: Using Depth of Thought as a Difficulty Signal for Tuning LLMscited by 3
  17. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  18. 2025On Theoretical Interpretations of Concept-Based In-Context Learningcited by 2
  19. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  20. 2025CUFG: Curriculum Unlearning Guided by the Forgetting Gradientcited by 1
  21. 2025On the Relationship Between the Choice of Representation and In-Context Learningcited by 1
  22. 2026MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learningcited by 1
  23. 2026Learning to Select Visual In-Context Demonstrationscited by 1
  24. 2026Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamicscited by 1
  25. 2024Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Modelscited by 0
  26. 2024Sparse Linear Bandits with Blocking Constraintscited by 0
  27. 2025Exploring Imbalanced Annotations for Effective In-Context Learningcited by 0
  28. 2025Meta-Learning Transformers to Improve In-Context Generalizationcited by 0
  29. 2025NeuroDx-LM: A Clinical Large-Scale Model for EEG-based Neurological Disorder Detectioncited by 0
  30. 2025A global log for medical AIcited by 0
  31. 2025Large Language Model Aided Birt-Hogg-Dube Syndrome Diagnosis with Multimodal Retrieval-Augmented Generationcited by 0
  32. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspectivecited by 0
  33. 2026UserGPT Technical Reportcited by 0
  34. 2026Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Tracescited by 0
  35. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0