# CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

arXiv: https://arxiv.org/abs/2609.40048 · HTML: https://mlsanity.com/paper/2609.40048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 48 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 12 of 12)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [Vidi: Large Multimodal Models for Video Understanding and Editing](https://mlsanity.com/paper/2504.15681/) (2025)
- [SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems](https://mlsanity.com/paper/2606.01314/) (2026)
- [EvoGround: Self-Evolving Video Agents for Video Temporal Grounding](https://mlsanity.com/paper/2605.13803/) (2026)
- [Vidi2.5: Large Multimodal Models for Video Understanding and Creation](https://mlsanity.com/paper/2511.19529/) (2025)
- [Natural-Language Temporal Grounding in Hour-Long Videos is a Search Problem: A Benchmark and Empirical Decomposition](https://mlsanity.com/paper/2606.12300/) (2026)
- [Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence](https://mlsanity.com/paper/2608.12743/) (2026)
- [Conditional Multi-Event Temporal Grounding in Long-Form Video](https://mlsanity.com/paper/2606.15320/) (2026)

## Abstract (arXiv)

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.