# Re-ReST: Reflection-Reinforced Self-Training for Language Agents

arXiv: https://arxiv.org/abs/2406.01495 · HTML: https://mlsanity.com/paper/2406.01495/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)

## Built on by (most cited first, 29 of 29)

- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [AgentRx: Diagnosing AI Agent Failures from Execution Trajectories](https://mlsanity.com/paper/2602.02475/) (2026)
- [A Survey on the Optimization of Large Language Model-based Agents](https://mlsanity.com/paper/2503.12434/) (2025)
- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation](https://mlsanity.com/paper/2506.04614/) (2025)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection](https://mlsanity.com/paper/2505.16475/) (2025)
- [Bootstrapping Task Spaces for Self-Improvement](https://mlsanity.com/paper/2509.04575/) (2025)
- [From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment](https://mlsanity.com/paper/2505.16610/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Interactive Learning for LLM Reasoning](https://mlsanity.com/paper/2509.26306/) (2025)
- [S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?](https://mlsanity.com/paper/2608.31100/) (2026)
- [AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs](https://mlsanity.com/paper/2507.18584/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Self-evolving LLM agents with in-distribution Optimization](https://mlsanity.com/paper/2606.07367/) (2026)
- [Masked Distillation: Internalizing the Chain-of-Thought in Language Models](https://mlsanity.com/paper/2607.22629/) (2026)
- [Meta-Reflection: A Feedback-Free Reflection Learning Framework](https://mlsanity.com/paper/2412.13781/) (2024)
- [MagicAgent: Towards Generalized Agent Planning](https://mlsanity.com/paper/2602.19000/) (2026)
- [Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment](https://mlsanity.com/paper/2604.19548/) (2026)
- [Active Inference for Self-Organizing Multi-LLM Systems: A Bayesian Thermodynamic Approach to Adaptation](https://mlsanity.com/paper/2412.10425/) (2024)
- [Exploring the Reliability of Self-explanation and its Relationship with Classification in Language Model-driven Financial Analysis](https://mlsanity.com/paper/2503.15985/) (2025)
- [Human-Inspired Continuous Learning of Internal Reasoning Processes: Learning How to Think for Adaptive AI Systems](https://mlsanity.com/paper/2602.11516/) (2026)
- [ReCAPA: Hierarchical Predictive Correction to Mitigate Cascading Failures](https://mlsanity.com/paper/2604.21232/) (2026)
- [AIPO: Learning to Reason from Active Interaction](https://mlsanity.com/paper/2605.08401/) (2026)
- [Exploration-Driven Optimization for Test-Time Large Language Model Reasoning](https://mlsanity.com/paper/2605.09853/) (2026)
- [ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation](https://mlsanity.com/paper/2609.39306/) (2026)

## Abstract (arXiv)

Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impractical. In this paper, we investigate the use of self-training in language agents, which can generate supervision from the agent itself, offering a promising alternative without relying on human or stronger model demonstrations. Self-training, however, requires high-quality model-generated samples, which are hard to obtain for challenging language agent tasks. To address this, we present Reflection-Reinforced Self-Training (Re-ReST), which uses a textitreflector to refine low-quality generated samples during self-training. The reflector takes the agent's output and feedback from an external environment (e.g., unit test results in code generation) to produce improved samples. This technique enhances the quality of inferior samples and efficiently enriches the self-training dataset with higher-quality samples. We conduct extensive experiments on open-source language agents across tasks, including multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation. The results demonstrate the effectiveness of self-training and Re-ReST in language agent tasks, with self-training improving baselines by 7.6% on HotpotQA and 28.4% on AlfWorld, and Re-ReST further boosting performance by 2.0% and 14.1%, respectively. Our studies also confirm the efficiency of using a reflector to generate high-quality samples for self-training. Moreover, we demonstrate a method to employ reflection during inference without ground-truth feedback, addressing the limitation of previous reflection work. Our code is released at https://github.com/PlusLabNLP/Re-ReST.