mlsanityData to 7 Oct 2026

Teaching Language Models to Self-Improve through Interactive Demonstrations

arXiv 2310.13522 ↗First posted 20 Oct 2023cs.CLcited by 26

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research. However, this ability has been shown to be absent and difficult to learn for smaller models, thus widening the performance gap between state-of-the-art LLMs and more cost-effective and faster ones. To reduce this gap, we introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability, and show that our approach can improve a LLaMA-7b's performance on math and reasoning tasks by up to 7.13%. In contrast to prior work, we achieve this by using the smaller model to interact with LLMs to collect feedback and improvements on its own generations. We then replay this experience to train the small model. Our experiments on four math and reasoning datasets show that the interactive experience of learning from and correcting its own mistakes is crucial for small models to improve their performance.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 42 of its 59 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

42 of 59 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
30 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2013Do Deep Nets Really Need to be Deep?cited by 466
  11. 2022Self-critiquing models for assisting human evaluatorscited by 372
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  13. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  14. 2022Large Language Models Are Reasoning Teacherscited by 318
  15. 2022Generating Sequences by Learning to Self-Correctcited by 271
  16. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  17. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  18. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  19. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  20. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  21. 2023Training Language Models with Language Feedback at Scalecited by 125
  22. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  23. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  24. 2021Probabilistic Margins for Instance Reweighting in Adversarial Trainingcited by 57
  25. 2022Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revisioncited by 46
  26. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  27. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  28. 2021Think about it! Improving defeasible reasoning by first modeling the question scenariocited by 20
  29. 2019Gradient Boosting Machine: A Surveycited by 7
  30. 2022One-vs-the-Rest Loss to Focus on Important Samples in Adversarial Trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 12; 2026: 2.

26 papers.

Sort:

By area:

  1. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  2. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  3. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  4. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  5. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 41
  6. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verificationcited by 23
  7. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 17
  8. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  9. 2024How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responsescited by 12
  10. 2024Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-upcited by 10
  11. 2024Teaching Language Models to Self-Improve by Learning from Language Feedbackcited by 9
  12. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
14 more
  1. 2024LIONs: An Empirically Optimized Approach to Align Language Modelscited by 7
  2. 2024SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Modelscited by 6
  3. 2025Self-Improving Loops for Visual Robotic Planningcited by 6
  4. 2025Entropy-Based Adaptive Weighting for Self-Trainingcited by 5
  5. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
  6. 2024Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedbackcited by 4
  7. 2025ConFit v2: Improving Resume-Job Matching using Hypothetical Resume Embedding and Runner-Up Hard-Negative Miningcited by 4
  8. 2026What If We Allocate Test-Time Compute Adaptively?cited by 4
  9. 2025Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Modelscited by 3
  10. 2025Self-Reflective Generation at Test Timecited by 3
  11. 2024CorrectionLM: Self-Corrections with SLM for Dialogue State Trackingcited by 1
  12. 2025CoLa: Learning to Interactively Collaborate with Large Language Modelscited by 0
  13. 2025Critique-Guided Distillation for Robust Reasoning via Refinementcited by 0
  14. 2026SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
  2. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  3. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  4. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
  5. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  6. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
  7. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
  8. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap