# Teaching Language Models to Self-Improve through Interactive Demonstrations

arXiv: https://arxiv.org/abs/2310.13522 · HTML: https://mlsanity.com/paper/2310.13522/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Not All Samples Are Created Equal: Deep Learning with Importance Sampling](https://mlsanity.com/paper/1803.00942/) (2018)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)

## Built on by (most cited first, 26 of 26)

- [Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains](https://mlsanity.com/paper/2501.05707/) (2025)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning](https://mlsanity.com/paper/2410.02052/) (2024)
- [Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models](https://mlsanity.com/paper/2402.12563/) (2024)
- [SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning](https://mlsanity.com/paper/2502.04780/) (2025)
- [TOOLVERIFIER: Generalization to New Tools via Self-Verification](https://mlsanity.com/paper/2402.14158/) (2024)
- [ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement](https://mlsanity.com/paper/2410.02108/) (2024)
- [Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agents](https://mlsanity.com/paper/2506.00320/) (2025)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up](https://mlsanity.com/paper/2410.12323/) (2024)
- [Teaching Language Models to Self-Improve by Learning from Language Feedback](https://mlsanity.com/paper/2406.07168/) (2024)
- [Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs](https://mlsanity.com/paper/2503.23219/) (2025)
- [LIONs: An Empirically Optimized Approach to Align Language Models](https://mlsanity.com/paper/2407.06542/) (2024)
- [Self-Improving Loops for Visual Robotic Planning](https://mlsanity.com/paper/2506.06658/) (2025)
- [SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models](https://mlsanity.com/paper/2408.15565/) (2024)
- [ConFit v2: Improving Resume-Job Matching using Hypothetical Resume Embedding and Runner-Up Hard-Negative Mining](https://mlsanity.com/paper/2502.12361/) (2025)
- [Entropy-Based Adaptive Weighting for Self-Training](https://mlsanity.com/paper/2503.23913/) (2025)
- [Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future](https://mlsanity.com/paper/2508.06026/) (2025)
- [Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedback](https://mlsanity.com/paper/2406.17873/) (2024)
- [What If We Allocate Test-Time Compute Adaptively?](https://mlsanity.com/paper/2602.01070/) (2026)
- [Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models](https://mlsanity.com/paper/2501.07892/) (2025)
- [Self-Reflective Generation at Test Time](https://mlsanity.com/paper/2510.02919/) (2025)
- [Critique-Guided Distillation for Robust Reasoning via Refinement](https://mlsanity.com/paper/2505.11628/) (2025)
- [CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking](https://mlsanity.com/paper/2410.18209/) (2024)
- [CoLa: Learning to Interactively Collaborate with Large Language Models](https://mlsanity.com/paper/2504.02965/) (2025)
- [SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization](https://mlsanity.com/paper/2601.22491/) (2026)

## Abstract (arXiv)

The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research. However, this ability has been shown to be absent and difficult to learn for smaller models, thus widening the performance gap between state-of-the-art LLMs and more cost-effective and faster ones. To reduce this gap, we introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability, and show that our approach can improve a LLaMA-7b's performance on math and reasoning tasks by up to 7.13%. In contrast to prior work, we achieve this by using the smaller model to interact with LLMs to collect feedback and improvements on its own generations. We then replay this experience to train the small model. Our experiments on four math and reasoning datasets show that the interactive experience of learning from and correcting its own mistakes is crucial for small models to improve their performance.