# CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

arXiv: https://arxiv.org/abs/2604.15840 · HTML: https://mlsanity.com/paper/2604.15840/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 28 of 28)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning](https://mlsanity.com/paper/2504.02546/) (2025)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training](https://mlsanity.com/paper/2501.11425/) (2025)
- [Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents](https://mlsanity.com/paper/2502.11357/) (2025)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories](https://mlsanity.com/paper/2410.07706/) (2024)
- [DeepAnalyze: Agentic Large Language Models for Autonomous Data Science](https://mlsanity.com/paper/2510.16872/) (2025)
- [STeCa: Step-level Trajectory Calibration for LLM Agent Learning](https://mlsanity.com/paper/2502.14276/) (2025)
- [Demystifying Reinforcement Learning in Agentic Reasoning](https://mlsanity.com/paper/2510.11701/) (2025)
- [Scaling Synthetic Task Generation for Agents via Exploration](https://mlsanity.com/paper/2509.25047/) (2025)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO](https://mlsanity.com/paper/2505.11595/) (2025)
- [A Comprehensive Survey on Rare Event Prediction](https://mlsanity.com/paper/2309.11356/) (2023)
- [Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking](https://mlsanity.com/paper/2505.20023/) (2025)
- [LIMI: Less is More for Agency](https://mlsanity.com/paper/2509.17567/) (2025)
- [ComPO: Preference Alignment via Comparison Oracles](https://mlsanity.com/paper/2505.05465/) (2025)
- [LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback](https://mlsanity.com/paper/2506.02298/) (2025)
- [An Empirical Study on How Video-LLMs Answer Video Questions](https://mlsanity.com/paper/2508.15360/) (2025)
- [Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning](https://mlsanity.com/paper/2511.08246/) (2025)

## Built on by (most cited first, 23 of 23)

- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks](https://mlsanity.com/paper/2608.01964/) (2026)
- [SEAL: Synergistic Co-Evolution of Agents and Learning Environments](https://mlsanity.com/paper/2605.24426/) (2026)
- [Escher-Loop: Mutual Evolution by Closed-Loop Self-Referential Optimization](https://mlsanity.com/paper/2604.23472/) (2026)
- [Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents](https://mlsanity.com/paper/2605.10832/) (2026)
- [Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2606.12634/) (2026)
- [Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Models](https://mlsanity.com/paper/2607.16900/) (2026)
- [Environment Evolution for Terminal Agents](https://mlsanity.com/paper/2609.04128/) (2026)
- [RISE: Reliable Improvement in Self-Evolving Vision-Language Models](https://mlsanity.com/paper/2605.20914/) (2026)
- [Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design](https://mlsanity.com/paper/2608.10299/) (2026)
- [EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2609.12459/) (2026)
- [D2Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2605.17037/) (2026)
- [Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue](https://mlsanity.com/paper/2608.10626/) (2026)
- [AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?](https://mlsanity.com/paper/2609.35025/) (2026)
- [ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization](https://mlsanity.com/paper/2610.00906/) (2026)
- [ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement](https://mlsanity.com/paper/2610.06347/) (2026)
- [Know What You Don't Know: Agentic Data Synthesis by Evolving Generation Space for Closed-Loop Learning](https://mlsanity.com/paper/2610.12593/) (2026)
- [Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval](https://mlsanity.com/paper/2610.14465/) (2026)
- [Heterogeneous Agentic Coevolution](https://mlsanity.com/paper/2610.16078/) (2026)
- [EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents](https://mlsanity.com/paper/2610.16523/) (2026)
- [Cowork-Gym: Synthesizing Interactive Work Environments at Scale](https://mlsanity.com/paper/2610.17670/) (2026)
- [ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolution](https://mlsanity.com/paper/2610.17718/) (2026)
- [Coupled Harness--Model Evolution for Scientific Equation Discovery](https://mlsanity.com/paper/2610.17740/) (2026)

## Abstract (arXiv)

Reinforcement learning for LLM agents is typically conducted on a static data distribution, which fails to adapt to the agent's evolving behavior and leads to poor coverage of complex environment interactions. To address these challenges, we propose CoEvolve, an agent-data mutual evolution framework that enables LLM agents to improve through closed-loop, interaction-driven training. Specifically, CoEvolve extracts feedback signals such as forgetting and uncertainty from rollout trajectories to identify failure-prone interaction patterns, and utilizes them to guide LLM-based task synthesis. The synthesized tasks are validated through environment interaction and utilized to update the data distribution, enabling joint adaptation of the agent and its data. Extensive experiments on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.