# LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

arXiv: https://arxiv.org/abs/2607.18110 · HTML: https://mlsanity.com/paper/2607.18110/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 52 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 30 of 47)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Critique-out-Loud Reward Models](https://mlsanity.com/paper/2408.11791/) (2024)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback](https://mlsanity.com/paper/2506.03106/) (2025)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training](https://mlsanity.com/paper/2606.30406/) (2026)
- [CRISP: Compressed Reasoning via Iterative Self-Policy Distillation](https://mlsanity.com/paper/2603.05433/) (2026)

## Built on by (most cited first, 4 of 4)

- [Flux-OPD: On-Policy Distillation with Evolving Contexts](https://mlsanity.com/paper/2607.28022/) (2026)
- [Learning to Coach for Experiential Learning](https://mlsanity.com/paper/2609.15851/) (2026)
- [CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning](https://mlsanity.com/paper/2610.17733/) (2026)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.