# In-Context Principle Learning from Mistakes

arXiv: https://arxiv.org/abs/2402.05403 · HTML: https://mlsanity.com/paper/2402.05403/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)

## Built on by (most cited first, 30 of 44)

- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [LLMs Are In-Context Bandit Reinforcement Learners](https://mlsanity.com/paper/2410.05362/) (2024)
- [No Need for Explanations: LLMs can implicitly learn from mistakes in-context](https://mlsanity.com/paper/2502.08550/) (2025)
- [Large Language Models are Contrastive Reasoners](https://mlsanity.com/paper/2403.08211/) (2024)
- [Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information](https://mlsanity.com/paper/2410.04463/) (2024)
- [Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs](https://mlsanity.com/paper/2410.20749/) (2024)
- [Fact-Augmented Lookahead Planning for LLM Agents](https://mlsanity.com/paper/2506.09171/) (2025)
- [Can Large Language Models Solve Robot Routing?](https://mlsanity.com/paper/2403.10795/) (2024)
- [AdaSwitch: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learning](https://mlsanity.com/paper/2410.13181/) (2024)
- [PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling](https://mlsanity.com/paper/2502.01925/) (2025)
- [Diverse Inference and Verification for Advanced Reasoning](https://mlsanity.com/paper/2502.09955/) (2025)
- [Retrieved In-Context Principles from Previous Mistakes](https://mlsanity.com/paper/2407.05682/) (2024)
- [WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation](https://mlsanity.com/paper/2503.19065/) (2025)
- [Training-free LLM Verification via Recycling Few-shot Examples](https://mlsanity.com/paper/2506.17251/) (2025)
- [Streamlining evidence based clinical recommendations with large language models](https://mlsanity.com/paper/2505.10282/) (2025)
- [Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Cases](https://mlsanity.com/paper/2508.12031/) (2025)
- [Localizing and Correcting Errors for LLM-based Planners](https://mlsanity.com/paper/2602.00276/) (2026)
- [SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation](https://mlsanity.com/paper/2506.00319/) (2025)
- [Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptation](https://mlsanity.com/paper/2512.11485/) (2025)
- [POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models](https://mlsanity.com/paper/2406.03843/) (2024)
- [Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting](https://mlsanity.com/paper/2408.09365/) (2024)
- [Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data](https://mlsanity.com/paper/2601.11038/) (2026)
- [FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification](https://mlsanity.com/paper/2604.03245/) (2026)
- [Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks](https://mlsanity.com/paper/2606.27147/) (2026)

## Abstract (arXiv)

In-context learning (ICL, also known as few-shot prompting) has been the standard method of adapting LLMs to downstream tasks, by learning from a few input-output examples. Nonetheless, all ICL-based approaches only learn from correct input-output pairs. In this paper, we revisit this paradigm, by learning more from the few given input-output examples. We introduce Learning Principles (LEAP): First, we intentionally induce the model to make mistakes on these few examples; then we reflect on these mistakes, and learn explicit task-specific "principles" from them, which help solve similar problems and avoid common mistakes; finally, we prompt the model to answer unseen test questions using the original few-shot examples and these learned general principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning, and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and Claude-2.1. For example, LEAP improves over the standard few-shot prompting using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does not require any more input or examples than the standard few-shot prompting settings.