# Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

arXiv: https://arxiv.org/abs/2607.17558 · HTML: https://mlsanity.com/paper/2607.17558/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 18 of 18)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning](https://mlsanity.com/paper/2511.16043/) (2025)
- [SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents](https://mlsanity.com/paper/2508.02085/) (2025)
- [s3: You Don't Need That Much Data to Train a Search Agent via RL](https://mlsanity.com/paper/2505.14146/) (2025)
- [Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data](https://mlsanity.com/paper/2602.21320/) (2026)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [Can Question Rewriting Help Conversational Question Answering?](https://mlsanity.com/paper/2204.06239/) (2022)
- [SE-Search: Self-Evolving Search Agent via Memory and Dense Reward](https://mlsanity.com/paper/2603.03293/) (2026)
- [From Knowledge to Noise: CTIM-Rover and the Pitfalls of Episodic Memory in Software Engineering Agents](https://mlsanity.com/paper/2505.23422/) (2025)
- [A Study on Leveraging Search and Self-Feedback for Agent Reasoning](https://mlsanity.com/paper/2502.12094/) (2025)
- [Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents](https://mlsanity.com/paper/2603.09203/) (2026)
- [SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training](https://mlsanity.com/paper/2603.18079/) (2026)

## Built on by (most cited first, 3 of 3)

- [Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09263/) (2026)
- [Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2609.37132/) (2026)
- [More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.19789/) (2026)

## Abstract (arXiv)

On-policy self-distillation (OPSD) offers a promising approach for training large language models without relying on a separate teacher model. However, its effectiveness on complex agentic tasks remains largely unexplored. In this work, we instantiate Feedback-Augmented Self-Distillation (FA-SD), a self-distillation algorithm for agentic search that leverages successful demonstrations as privileged information. We identify that models can rely on recurring reasoning-and-search output templates, producing trajectories that appear diverse but are largely agnostic to the input question, making the KL-based self-distillation signal uninformative. We term this phenomenon decoding collapse, a failure mode that can be missed by existing evaluation metrics. To understand its underlying cause, we show that although the self-teacher achieves stronger performance, learning remains inherently unstable due to inconsistent supervision signals. We further decompose this inconsistency into model inconsistency and prompt inconsistency, and show that the latter can significantly degrade the quality of the supervision signal, limiting the effectiveness of self-teacher learning. To mitigate this inconsistency, we introduce an exponential moving average (EMA) teacher to stabilize the self-teacher and provide more consistent supervision signals. Although the EMA teacher requires a warm-up phase during which performance may temporarily regress, it ultimately improves model performance by providing more stable supervision.