# ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection

arXiv: https://arxiv.org/abs/2505.16475 · HTML: https://mlsanity.com/paper/2505.16475/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 36)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [MALT: Improving Reasoning with Multi-Agent LLM Training](https://mlsanity.com/paper/2412.01928/) (2024)
- [Language Model Self-improvement by Reinforcement Learning Contemplation](https://mlsanity.com/paper/2305.14483/) (2023)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints](https://mlsanity.com/paper/2410.06458/) (2024)
- [Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks](https://mlsanity.com/paper/2410.04055/) (2024)
- [Re-ReST: Reflection-Reinforced Self-Training for Language Agents](https://mlsanity.com/paper/2406.01495/) (2024)
- [I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm](https://mlsanity.com/paper/2408.08072/) (2024)
- [Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning](https://mlsanity.com/paper/2310.11716/) (2023)
- [Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.18248/) (2024)

## Built on by (most cited first, 13 of 13)

- [Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs](https://mlsanity.com/paper/2508.19594/) (2025)
- [The AI Hippocampus: How Far are We From Human Memory?](https://mlsanity.com/paper/2601.09113/) (2026)
- [RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling](https://mlsanity.com/paper/2506.08672/) (2025)
- [Retrospective Progress-Aware Self-Refinement for LLM Agent Training](https://mlsanity.com/paper/2606.14302/) (2026)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility](https://mlsanity.com/paper/2605.29229/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [MetaCrit: A Critical Thinking Framework for Self-Regulated LLM Reasoning](https://mlsanity.com/paper/2507.15015/) (2025)
- [PACE-RAG: Patient-Aware Contextual and Evidence-Constrained RAG for Clinical Drug Recommendation](https://mlsanity.com/paper/2603.17356/) (2026)
- [LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts](https://mlsanity.com/paper/2602.14060/) (2026)
- [SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptation](https://mlsanity.com/paper/2605.20189/) (2026)
- [Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes](https://mlsanity.com/paper/2609.19156/) (2026)
- [MENTOR: Learning from Failure with On-Policy Pedagogical Intervention](https://mlsanity.com/paper/2610.13594/) (2026)

## Abstract (arXiv)

We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs' reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis of the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.