mlsanityData to 13 Oct 2026

In-Context Principle Learning from Mistakes

arXiv 2402.05403 ↗First posted 8 Feb 2024cs.CL cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL, also known as few-shot prompting) has been the standard method of adapting LLMs to downstream tasks, by learning from a few input-output examples. Nonetheless, all ICL-based approaches only learn from correct input-output pairs. In this paper, we revisit this paradigm, by learning more from the few given input-output examples. We introduce Learning Principles (LEAP): First, we intentionally induce the model to make mistakes on these few examples; then we reflect on these mistakes, and learn explicit task-specific "principles" from them, which help solve similar problems and avoid common mistakes; finally, we prompt the model to answer unseen test questions using the original few-shot examples and these learned general principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning, and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and Claude-2.1. For example, LEAP improves over the standard few-shot prompting using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does not require any more input or examples than the standard few-shot prompting settings.

Key facts. Cited by 44 papers in the map (data through 13 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

43 of 55 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
31 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  5. 2023A Survey of Large Language Modelscited by 2,559
  6. 2022Emergent Abilities of Large Language Modelscited by 2,499
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,229
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,445
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,404
  13. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,191
  14. 2023PaLM 2 Technical Reportcited by 1,162
  15. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 983
  16. 2023Teaching Large Language Models to Self-Debugcited by 972
  17. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 913
  18. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  19. 2023Large Language Models as Optimizerscited by 842
  20. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  21. 2022Large Language Models Can Self-Improvecited by 659
  22. 2021MetaICL: Learning to Learn In Contextcited by 452
  23. 2023Larger language models do in-context learning differentlycited by 387
  24. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  25. 2023Challenges and Applications of Large Language Modelscited by 253
  26. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  27. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  28. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  29. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  30. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  31. 2023Contrastive Chain-of-Thought Promptingcited by 52

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 16; 2025: 16; 2026: 12.

44 papers.

Sort:

By area:

  1. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 295
  2. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 269
  3. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 214
  4. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 164
  5. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 100
  6. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  7. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 38
  8. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 28
  9. 2024Large Language Models are Contrastive Reasonerscited by 11
  10. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 11
  11. 2024Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Informationcited by 10
  12. 2024Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMscited by 9
32 more
  1. 2025Fact-Augmented Lookahead Planning for LLM Agentscited by 8
  2. 2024Can Large Language Models Solve Robot Routing?cited by 6
  3. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  4. 2025Diverse Inference and Verification for Advanced Reasoningcited by 6
  5. 2024Retrieved In-Context Principles from Previous Mistakescited by 5
  6. 2024AdaSwitch: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learningcited by 5
  7. 2025WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generationcited by 5
  8. 2025Training-free LLM Verification via Recycling Few-shot Examplescited by 4
  9. 2025Streamlining evidence based clinical recommendations with large language modelscited by 3
  10. 2026Localizing and Correcting Errors for LLM-based Plannerscited by 3
  11. 2025SkillVerse : Assessing and Enhancing LLMs with Tree Evaluationcited by 2
  12. 2025Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Casescited by 2
  13. 2025Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptationcited by 2
  14. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Modelscited by 1
  15. 2024Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Promptingcited by 1
  16. 2026Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Datacited by 1
  17. 2026Safe Autoregressive Image Generation with Iterative Self-Improving Codebookscited by 1
  18. 2024Can we only use guideline instead of shot in prompt?cited by 0
  19. 2024A Novel Task-Driven Method with Evolvable Interactive Agents Using Event Trees for Enhanced Emergency Decision Supportcited by 0
  20. 2025CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theorycited by 0
  21. 2025A Training-free LLM Framework with Interaction between Contextually Related Subtasks in Solving Complex Taskscited by 0
  22. 2025Retrieval Enhanced Feedback via In-context Neural Error-bookcited by 0
  23. 2025Reasoning Shapes Alignment: Investigating Cultural Alignment in Large Reasoning Models with Cultural Normscited by 0
  24. 2026TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thoughtcited by 0
  25. 2026A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvementcited by 0
  26. 2026FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verificationcited by 0
  27. 2026Supplement Generation Training for Enhancing Agentic Task Performancecited by 0
  28. 2026Smarter by the Moment: Environment-Driven Dynamic Policies for Continual LLM Improvementcited by 0
  29. 2026The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scalecited by 0
  30. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0
  31. 2026SEER: Self-Evolving Event Reasoning and Retrieval for Time Series Forecastingcited by 0
  32. 2026Verb-ICL: Rethinking In-Context Learning for Structured Predictioncited by 0